Original:  S TATISTICAL V ERIFICATION OF N EURAL N ETWORKS
Revised: A S TATISTICAL A PPROACH TO A SSESSING N EURAL N ETWORK R OBUSTNESS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a new approach to  neural network veriﬁcation based on estimating the proportion of inputs for which a property is violated.
Revised: We present a new approach to assessing the robustness of neural networks based on estimating the proportion of inputs for which a property is violated.

Original: Speciﬁcally, we estimate the probability of the event that the property is violated under an input model.
Revised: Speciﬁcally, we estimate the probability of the event that the property is violated under an input model.

Original: This permits classic veriﬁcation as a special case, for which one considers only the question of whether this expectation is exactly zero or not. When the property can be violated, our approach provides an informative notion of how robust the network is, rather than just the conventional assertion that the network is not veriﬁable.
Revised: Our approach critically varies from the formal veriﬁcation framework in that when the property can be violated, it provides an informative notion of how robust the network is, rather than just the conventional assertion that the network is not veriﬁable.

Original: Furthermore, it provides an ability to scale to larger networks thanclassical formal veriﬁcation approaches.
Revised: Furthermore, it provides an ability to scale to larger networks than formal veriﬁcation approaches.

Revised: Though the framework still provides a formal guarantee of satisﬁability whenever it successfully ﬁnds one or more violations, these advantages do come at the cost of only providing a statistical estimate of unsatisﬁability whenever no violation is found.

Original: Key to achieving this is an adaptation of multi-level splitting, a Monte Carlo approach for estimating the probability of rare events, to our statistical veriﬁcation framework.
Revised: Key to the practical success of our approach is an adaptation of multi-level splitting, a Monte Carlo approach for estimating the probability of rare events, to our statistical robustness framework.

Original: We demonstrate that our approach is able to emulate existing veriﬁcation procedures on benchmark problems, while scaling to larger networks and providing reliable additional information in the form of accurate estimates of the violation probability.
Revised: We demonstrate that our approach is able to emulate formal veriﬁcation procedures on benchmark problems, while scaling to larger networks and providing reliable additional information in the form of accurate estimates of the violation probability.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The robustness of deep neural networks must be guaranteed in mission-critical applications where their failure could have severe real-world implications.
Revised: The robustness of deep neural networks must be guaranteed in mission-critical applications where their failure could have severe real-world implications.

Original: This motivates the study of neural network veriﬁcation, in which one wishes to assert whether certain inputs in a given subdomain of the network might lead to important properties being violated (Zakrzewski, 2001; Bunel et al., 2018).
Revised: This motivates the study of neural network veriﬁcation, in which one wishes to assert whether certain inputs in a given subdomain of the network might lead to important properties being violated (Zakrzewski, 2001; Bunel et al., 2018).

Original: For example, in a classiﬁcation task, one might want to ensure that small perturbations of the inputs do not lead to incorrect class labels being predicted (Szegedy et al., 2013; Goodfellow et al., 2015).
Revised: For example, in a classiﬁcation task, one might want to ensure that small perturbations of the inputs do not lead to incorrect class labels being predicted (Szegedy et al., 2013; Goodfellow et al., 2015).

Original: The classic approach to such veriﬁcation has focused on answering the binary question of whether there exist any counterexamples that violate the property of interest.
Revised: The classic approach to such veriﬁcation has focused on answering the binary question of whether there exist any counterexamples that violate the property of interest.

Original: We argue that this approach has two major drawbacks.
Revised: We argue that this approach has two major drawbacks.

Original: Firstly, it provides no notion of how robust a network is whenever a counterexample can be found.
Revised: Firstly, it provides no notion of how robust a network is whenever a counterexample can be found.

Original: Secondly, it creates a computational problem whenever no counterexamples exist, as formally verifying this can be very costly and does not currently scale to the size of networks used in many applications.
Revised: Secondly, it creates a computational problem whenever no counterexamples exist, as formally verifying this can be very costly and does not currently scale to the size of networks used in many applications.

Original: To give a demonstrative example, consider a neural network for classifying objects in the path of an autonomous vehicle.
Revised: To give a demonstrative example, consider a neural network for classifying objects in the path of an autonomous vehicle.

Original: It will almost certainly be infeasible to train such a network that is perfectly robust to misclassiﬁcation.
Revised: It will almost certainly be infeasible to train such a network that is perfectly robust to misclassiﬁcation.

Original: Furthermore, because the network will most likely need to be of signiﬁcant size to be effective, it is unlikely to be tractable to formally verify the network is perfectly robust, even if such a network exists.
Revised: Furthermore, because the network will most likely need to be of signiﬁcant size to be effective, it is unlikely to be tractable to formally verify the network is perfectly robust, even if such a network exists.

Original: Despite this, it is still critically important to assess the robustness of the network, so that manufacturers can decide whether it is safe to deploy.
Revised: Despite this, it is still critically important to assess the robustness of the network, so that manufacturers can decide whether it is safe to deploy.

Original: To address the shortfalls of the classic approach, we develop a new measure of intrinsic robustness of neural networks based on the probability that a property is violated under an input distribution model.
Revised: To address the shortfalls of the classic approach, we develop a new measure of intrinsic robustness of neural networks based on the probability that a property is violated under an input distribution model.

Original: Our measure is based on two key insights.
Revised: Our measure is based on two key insights.

Original: The ﬁrst is that for many, if not most, applications, full formal veriﬁcation is neither necessary nor realistically achievable, such that one actually desires a notion of how robust a network is to a set of inputs, not just a binary answer as to whether it is robust or not.
Revised: The ﬁrst is that for many, if not most, applications, full formal veriﬁcation is neither necessary nor realistically achievable, such that one actually desires a notion of how robust a network is to a set of inputs, not just a binary answer as to whether it is robust or not.

Original: The second is that most practical applications have some acceptable level of risk, such that it is sufﬁcient to show that the probability of a violation is below a certain threshold, rather than conﬁrm that this probability is exactly zero.
Revised: The second is that most practical applications have some acceptable level of risk, such that it is sufﬁcient to show that the probability of a violation is below a certain threshold, rather than conﬁrm that this probability is exactly zero.

Original: By providing a probability of violation, our approach is able to address the needs of applications such as our autonomous vehicle example.
Revised: By providing a probability of violation, our approach is able to address the needs of applications such as our autonomous vehicle example.

Original: If the network is not perfectly robust, it provides an explicit measure of exactly how robust the network is.
Revised: If the network is not perfectly robust, it provides an explicit measure of exactly how robust the network is.

Original: If the network is perfectly robust, it is still able to tractability assert that a violation event is “probably-unsatisﬁable”.
Revised: If the network is perfectly robust, it is still able to tractability assert that a violation event is “probably-unsatisﬁable”.

Original: That is it is able to statistically conclude that the violation probability is below some tolerance threshold to true zero, even for large networks for which formal veriﬁcation would not be possible.
Revised: That is it is able to statistically conclude that the violation probability is below some tolerance threshold to true zero, even for large networks for which formal veriﬁcation would not be possible.

Original: Calculating the probability of violation is still itself a computationally challenging task, corresponding to estimating the value of an intractable integral.
Revised: Calculating the probability of violation is still itself a computationally challenging task, corresponding to estimating the value of an intractable integral.

Original: In particular, in most cases, violations of the target property constitute (potentially extremely) rare events.
Revised: In particular, in most cases, violations of the target property constitute (potentially extremely) rare events.

Original: Consequently, the simple approach of constructing a direct Monte Carlo estimate by sampling from the input model and evaluating the property will be expensive and only viable when the event is relatively common.
Revised: Consequently, the simple approach of constructing a direct Monte Carlo estimate by sampling from the input model and evaluating the property will be expensive and only viable when the event is relatively common.

Original: To address this, we adapt an algorithm from the Monte Carlo literature, adaptive multi-level splitting (AMLS) (Guyader et al., 2011; Nowozin, 2015), to our network veriﬁcation setting.
Revised: To address this, we adapt an algorithm from the Monte Carlo literature, adaptive multi-level splitting (AMLS) (Guyader et al., 2011; Nowozin, 2015), to our network veriﬁcation setting.

Original: AMLS is explicitly designed for prediction of rare events and our adaptation means that we are able to reliably estimate the probability of violation, even when the true value is extremely small.
Revised: AMLS is explicitly designed for prediction of rare events and our adaptation means that we are able to reliably estimate the probability of violation, even when the true value is extremely small.

Original: Our resulting framework is easy to implement, scales linearly in the cost of the forward operation of the neural network, and is agnostic both to the network architecture and input model.
Revised: Our resulting framework is easy to implement, scales linearly in the cost of the forward operation of the neural network, and is agnostic both to the network architecture and input model.

Original: Assumptions such as piecewise linearity, Lipschitz continuity, or a speciﬁc network form are not required.
Revised: Assumptions such as piecewise linearity, Lipschitz continuity, or a speciﬁc network form are not required.

Original: Furthermore, it produces a diversity of samples which violate the property as a side-product.
Revised: Furthermore, it produces a diversity of samples which violate the property as a side-product.

Original: To summarize, our main contributions are:
Revised: To summarize, our main contributions are:

Original: • Reframing neural network veriﬁcation as the estimation of the probability of a violation, thereby providing a more informative robustness metric for non-veriﬁable networks; • Adaptation of the AMLS method to our veriﬁcation framework to allow the tractable estimation of our metric for large networks and rare events; • Validation of our approach on several models and datasets from the literature.
Revised: • Reframing neural network veriﬁcation as the estimation of the probability of a violation, thereby providing a more informative robustness metric for non-veriﬁable networks; • Adaptation of the AMLS method to our veriﬁcation framework to allow the tractable estimation of our metric for large networks and rare events; • Validation of our approach on several models and datasets from the literature.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: The literature on neural network robustness follows two main threads.
Revised: The literature on neural network robustness follows two main threads.

Original: In the optimization community, researchers seek to formally prove that a property holds for a neural network by framing it as a satisﬁability problem (Zakrzewski, 2001), which we refer to as the classical approach to veriﬁcation.
Revised: In the optimization community, researchers seek to formally prove that a property holds for a neural network by framing it as a satisﬁability problem (Zakrzewski, 2001), which we refer to as the classical approach to veriﬁcation.

Original: Such methods have only been successfully scaled beyond one hidden layer networks for piecewise linear networks (Cheng et al., 2017; Katz et al., 2017), and even then these solutions do not scale to, for example, common image classiﬁcation architectures with input dimensions in the hundreds, or apply to networks with nonlinear activation functions (Bunel et al., 2018).
Revised: Such methods have only been successfully scaled beyond one hidden layer networks for piecewise linear networks (Cheng et al., 2017; Katz et al., 2017), and even then these solutions do not scale to, for example, common image classiﬁcation architectures with input dimensions in the hundreds, or apply to networks with nonlinear activation functions (Bunel et al., 2018).

Original: Other work has sought approximate solutions in the same general framework but still does not scale to larger networks (Pulina & Tacchella, 2010; Xiang et al., 2018;
Revised: Other work has sought approximate solutions in the same general framework but still does not scale to larger networks (Pulina & Tacchella, 2010; Xiang et al., 2018;

Original: Huang et al., 2017b).
Revised: Huang et al., 2017b).

Original: As the problem is
Revised: As the problem is

Original: NP-hard (Katz et al., 2017), it is unlikely that an algorithm exists with runtime scaling polynomially in the number of network nodes.
Revised: NP-hard (Katz et al., 2017), it is unlikely that an algorithm exists with runtime scaling polynomially in the number of network nodes.

Original: In the deep learning community, research has focused on constructing and defending against adversarial attacks, and by estimating the robustness of networks to such attacks.
Revised: In the deep learning community, research has focused on constructing and defending against adversarial attacks, and by estimating the robustness of networks to such attacks.

Original: Weng et al.
Revised: Weng et al.

Original: (2018b) recently constructed a measure for robustness to adversarial attacks estimating a lower bound on the minimum adversarial distortion, that is the smallest perturbation required to create an adversarial example.
Revised: (2018b) recently constructed a measure for robustness to adversarial attacks estimating a lower bound on the minimum adversarial distortion, that is the smallest perturbation required to create an adversarial example.

Original: Though the approach scales to large networks, the estimate of the lower bound is often demonstratively incorrect: it is often higher than an upper bound on the minimum adversarial distortion (Goodfellow, 2018).
Revised: Though the approach scales to large networks, the estimate of the lower bound is often demonstratively incorrect: it is often higher than an upper bound on the minimum adversarial distortion (Goodfellow, 2018).

Original: Other drawbacks of the method are that it cannot be applied to networks that are not Lipschitz continuous, it requires an expensive gradient computation for each class per sample, does not produce adversarial examples, and cannot be applied to non-adversarial properties.
Revised: Other drawbacks of the method are that it cannot be applied to networks that are not Lipschitz continuous, it requires an expensive gradient computation for each class per sample, does not produce adversarial examples, and cannot be applied to non-adversarial proper- ties.

Original: The minimum adversarial distortion is also itself a somewhat unsatisfying metric for many applications, as it conveys little information about the prevalence of adversarial examples.
Revised: The minimum adversarial distortion is also itself a somewhat unsatisfying metric for many applications, as it conveys little information about the prevalence of adversarial examples.

Original: In other work spanning both communities (Gehr et al., 2018; Weng et al., 2018a; Wong & Kolter, 2018), researchers have relaxed the satisﬁability problem of classical veriﬁcation, and are able to produce certiﬁcates-of-robustness for some samples (but not all that are robust) by giving a lowerbound on the minimal adversarial distortion.
Revised: In other work spanning both communities (Gehr et al., 2018; Weng et al., 2018a; Wong & Kolter, 2018), researchers have relaxed the satisﬁability problem of classical veriﬁcation, and are able to produce certiﬁcates-of-robustness for some samples (but not all that are robust) by giving a lowerbound on the minimal adversarial distortion.

Original: Despite these methods scaling beyond formal veriﬁcation, we note that this is still a binary measure of robustness with limited informativeness.
Revised: Despite these methods scaling beyond formal veriﬁcation, we note that this is still a binary measure of robustness with limited informativeness.

Original:  M OTIVATING E XAMPLES
Revised:  M OTIVATING E XAMPLES

Original: To help elucidate our problem setting, we consider the A CAS X U dataset (Katz et al., 2017) from the formal veriﬁcation literature.
Revised: To help elucidate our problem setting, we consider the A CAS X U dataset (Katz et al., 2017) from the formal veriﬁcation literature.

Original: A neural network is trained to predict one of ﬁve correct steering decisions, such as “hard left,” “soft left,” etc., for an unmanned aircraft to avoid collision with a second aircraft.
Revised: A neural network is trained to predict one of ﬁve correct steering decisions, such as “hard left,” “soft left,” etc., for an unmanned aircraft to avoid collision with a second aircraft.

Original: The inputs x describe the positions, orientations, velocities, etc. of the two aircraft.
Revised: The inputs x describe the positions, orientations, velocities, etc. of the two aircraft.

Original: Ten interpretable properties are speciﬁed along with corresponding constraints on the inputs, for which violations correspond to events causing collisions.
Revised: Ten interpretable properties are speciﬁed along with corresponding constraints on the inputs, for which violations correspond to events causing collisions.

Original: Each of these properties is encoded in a function, s , such that it is violated when s ( x ) ≥ 0 .
Revised: Each of these properties is encoded in a function, s , such that it is violated when s ( x ) ≥ 0 .

Original: The formal veriﬁcation problem asks the question, “Does there exist an input x ∈ E ⊆ X in a constrained subset, E , of the domain such that the property is violated?”
Revised: The formal veriﬁcation problem asks the question, “Does there exist an input x ∈ E ⊆ X in a constrained subset, E , of the domain such that the property is violated?”

Original: If there exists a counterexample violating the property, we say that the property is satisﬁable ( SAT ), and otherwise, unsatisﬁable ( UNSAT ).
Revised: If there exists a counterexample violating the property, we say that the property is satisﬁable ( SAT ), and otherwise, unsatisﬁable ( UNSAT ).

Original: Another example is provided by adversarial properties from the deep learning literature on datasets such as MNIST .
Revised: Another example is provided by adversarial properties from the deep learning literature on datasets such as MNIST .

Original: Consider a neural network f θ ( x ) =
Revised: Consider a neural network f θ ( x ) =

Original: Softmax ( z ( x )) that classiﬁes images, x , into
Revised: Softmax ( z ( x )) that classiﬁes images, x , into

Original: C classes, where the output of f gives the probability of each class.
Revised: C classes, where the output of f gives the probability of each class.

Original: Let δ be a small perturbation in an l p -ball of radius (cid:15) , that is, (cid:107) δ (cid:107) p < (cid:15) .
Revised: Let δ be a small perturbation in an l p -ball of radius (cid:15) , that is, (cid:107) δ (cid:107) p < (cid:15) .

Original: Then x = x (cid:48)
Revised: Then x = x (cid:48)

Original: + δ is an adversarial example for x (cid:48) if arg max i z ( x ) i (cid:54) =
Revised: + δ is an adversarial example for x (cid:48) if arg max i z ( x ) i (cid:54) =

Original: arg max i
Revised: arg max i

Original: z
Revised: z

Original: ( x (cid:48) )
Revised: ( x (cid:48) )

Original: i , i.e. the perturbation changes the prediction.
Revised: i , i.e. the perturbation changes the prediction.

Original: Here, the property function is s ( x ) = max i (cid:54) =
Revised: Here, the property function is s ( x ) = max i (cid:54) =

Original: c ( z ( x ) i − z
Revised: c ( z ( x ) i − z

Original: ( x ) c ) , where c = arg max j z ( x (cid:48) )
Revised: ( x ) c ) , where c = arg max j z ( x (cid:48) )

Original: j and s ( x ) ≥ 0 indicates that x is an adversarial example.
Revised: j and s ( x ) ≥ 0 indicates that x is an adversarial example.

Original: Our approach subsumes adversarial properties as a speciﬁc case.
Revised: Our approach subsumes adversarial properties as a speciﬁc case.

Original:  R OBUSTNESS M ETRIC
Revised:  R OBUSTNESS M ETRIC

Original: The framework for our robustness metric is very general, requiring only a) a neural network f θ , b) a property function s ( x ; f, φ ) , and c) an input model p ( x ) .
Revised: The framework for our robustness metric is very general, requiring only a) a neural network f θ , b) a property function s ( x ; f, φ ) , and c) an input model p ( x ) .

Original: Together these deﬁne an integration problem, with the main practical challenge being the estimation of this integral.
Revised: Together these deﬁne an integration problem, with the main practical challenge being the estimation of this integral.

Original: Consequently, the method can be used for any neural network.
Revised: Consequently, the method can be used for any neural network.

Original: The only requirement is that we can evaluate the property function, which typically involves a forward pass of the neural network.
Revised: The only requirement is that we can evaluate the property function, which typically involves a forward pass of the neural network.

Original: The property function, s ( x ; f θ , φ ) , is a deterministic function of the input x , the trained network f θ , and problem speciﬁc parameters φ .
Revised: The property function, s ( x ; f θ , φ ) , is a deterministic function of the input x , the trained network f θ , and problem speciﬁc parameters φ .

Original: For instance, in the MNIST example, φ = arg max i f θ ( x (cid:48) )
Revised: For instance, in the MNIST example, φ = arg max i f θ ( x (cid:48) )

Original: i is the true output of the unperturbed input.
Revised: i is the true output of the unperturbed input.

Original: Informally, the property reﬂects how badly the network is performing with respect to a particular property.
Revised: Informally, the property reﬂects how badly the network is performing with respect to a particular property.

Original: More precisely, the event
Revised: More precisely, the event

Original: [Equation]
Revised: [Equation]

Original: represents the property being violated.
Revised: represents the property being violated.

Original: Predicting the occurrence of these, typically rare, events will be the focus of our work.
Revised: Predicting the occurrence of these, typically rare, events will be the focus of our work.

Original: We will omit the dependency on f θ and φ from here on for notional conciseness, noting that these are assumed to be ﬁxed and known for veriﬁcation problems.
Revised: We will omit the dependency on f θ and φ from here on for notional conciseness, noting that these are assumed to be ﬁxed and known for veriﬁcation problems.

Original: The input model, p ( x ) , is a distribution over the subset of the input domain that we are considering for counterexamples.
Revised: The input model, p ( x ) , is a distribution over the subset of the input domain that we are considering for counterexamples.

Original: For instance, for the MNIST example we could use p ( x ; x (cid:48) )
Revised: For instance, for the MNIST example we could use p ( x ; x (cid:48) )

Original: ∝( (cid:107) x − x (cid:48) (cid:107) p ≤ (cid:15) ) to consider uniform perturbations to the input around an l p -norm ball with radius (cid:15) .
Revised: ∝( (cid:107) x − x (cid:48) (cid:107) p ≤ (cid:15) ) to consider uniform perturbations to the input around an l p -norm ball with radius (cid:15) .

Original: More generally, the input model can be used to place restrictions on the input domain and potentially also to reﬂect that certain violations might be more damaging than others.
Revised: More generally, the input model can be used to place restrictions on the input domain and potentially also to reﬂect that certain violations might be more damaging than others.

Original: Together, the property function and input model specify the probability of failure through the integral
Revised: Together, the property function and input model specify the probability of failure through the integral

Original: [Equation]
Revised: [Equation]

Original: This integral forms our measure of robustness.
Revised: This integral forms our measure of robustness.

Original: The integral being equal to exactly zero corresponds to the classical notion of a formally veriﬁable network.
Revised: The integral being equal to exactly zero corresponds to the classical notion of a formally veriﬁable network.

Original: Critically though, it also provides a measure for how robust a non-formally-veriﬁable network is.
Revised: Critically though, it also provides a measure for how robust a non-formally-veriﬁable network is.

Original:  M ETRIC E STIMATION
Revised:  M ETRIC E STIMATION

Original: Our primary goal is to estimate (2) in order to obtain a measure of robustness.
Revised: Our primary goal is to estimate (2) in order to obtain a measure of robustness.

Original: Ideally, we also wish to generate example inputs which violate the property.
Revised: Ideally, we also wish to generate example inputs which violate the property.

Original: Unfortunately, the event E is typically very rare in veriﬁcation scenarios.
Revised: Unfortunately, the event E is typically very rare in veriﬁcation scenarios.

Original: Consequently, the estimating the integral directly using Monte Carlo,
Revised: Consequently, the estimating the integral directly using Monte Carlo,

Original: [Equation]
Revised: [Equation]

Original: is typically not feasible for real problems, requiring an impractically large number of samples to achieve a reasonable accuracy.
Revised: is typically not feasible for real problems, requiring an impractically large number of samples to achieve a reasonable accuracy.

Original: Even when E is not a rare event, we desire to estimate the probability using as few forward passes of the neural network as possible to reduce computation.
Revised: Even when E is not a rare event, we desire to estimate the probability using as few forward passes of the neural network as possible to reduce computation.

Original: Furthermore, the dimensionality of x is typically large for practical problems, such that it is essential to employ a method that scales well in dimensionality.
Revised: Furthermore, the dimensionality of x is typically large for practical problems, such that it is essential to employ a method that scales well in dimensionality.

Original: Consequently many of the methods commonly employed for such problems, such as the cross-entropy method (Rubinstein, 1997; De Boer et al., 2005), are inappropriate due to relying on importance sampling, which is well known to scale poorly.
Revised: Consequently many of the methods commonly employed for such problems, such as the cross-entropy method (Rubinstein, 1997; De Boer et al., 2005), are inappropriate due to relying on importance sampling, which is well known to scale poorly.

Original: As we will demonstrate empirically, a less well known but highly effective method from the statistics literature, adaptive multi-level splitting (AMLS) (Kahn & Harris, 1951; Guyader et al., 2011), can be readily adapted to address all the aforementioned computational challenges.
Revised: As we will demonstrate empirically, a less well known but highly effective method from the statistics literature, adaptive multi-level splitting (AMLS) (Kahn & Harris, 1951; Guyader et al., 2011), can be readily adapted to address all the aforementioned computational challenges.

Original: Speciﬁcally, AMLS is explicitly designed for estimating the probability of rare events and our adaptation is able to give highly accurate estimates even when the E is very rare.
Revised: Speciﬁcally, AMLS is explicitly designed for estimating the probability of rare events and our adaptation is able to give highly accurate estimates even when the E is very rare.

Original: Furthermore, as will be explained later, AMLS also allows the use of MCMC transitions, meaning that our approach is able to scale effectively in the dimensionality of x .
Revised: Furthermore, as will be explained later, AMLS also allows the use of MCMC transitions, meaning that our approach is able to scale effectively in the dimensionality of x .

Original: A further desirable property of AMLS is that it produces property-violating examples as a side product, namely, it produces samples from the distribution
Revised: A further desirable property of AMLS is that it produces property-violating examples as a side product, namely, it produces samples from the distribution

Original: [Equation]
Revised: [Equation]

Original: Such samples could, in theory, be used to perform robust learning, in a similar spirit to Goodfellow et al.
Revised: Such samples could, in theory, be used to perform robust learning, in a similar spirit to Goodfellow et al.

Original: (2015) and Madry et al.
Revised: (2015) and Madry et al.

Original: 5. M ULTI - LEVEL SPLITTING
Revised: 5. M ULTI - LEVEL SPLITTING

Original: Multi-level splitting (Kahn & Harris, 1951) divides the problem of predicting the probability of a rare event into several simpler ones.
Revised: Multi-level splitting (Kahn & Harris, 1951) divides the problem of predicting the probability of a rare event into several simpler ones.

Original: Speciﬁcally, we construct a sequence of intermediate targets,
Revised: Speciﬁcally, we construct a sequence of intermediate targets,

Original: [Equation]
Revised: [Equation]

Original: for levels,
Revised: for levels,

Original: −∞ =
Revised: −∞ =

Original: L 0 <
Revised: L 0 <

Original: L 1 < L 2 < · · · < L K = 0 , to bridge the gap between the input model p ( x ) and the target π ( x ) .
Revised: L 1 < L 2 < · · · < L K = 0 , to bridge the gap between the input model p ( x ) and the target π ( x ) .

Original: For any choice of the intermediate levels, we can now represent equation (2) through the following factorization,
Revised: For any choice of the intermediate levels, we can now represent equation (2) through the following factorization,

Original: [Equation]
Revised: [Equation]

Original: where P k (cid:44) E
Revised: where P k (cid:44) E

Original: X ∼ π k − 1 ( · [Equation]
Revised: X ∼ π k − 1 ( · ) (cid:2) 1 { s ( X ) ≥ L k } (cid:3) .

Original: Provided consecutive levels are sufﬁciently close, we will be able to reliably estimate each P k by making use of the samples from one level to initialize the estimation of the next.
Revised: Provided consecutive levels are sufﬁciently close, we will be able to reliably estimate each P k by making use of the samples from one level to initialize the estimation of the next.

Original: Our approach starts by ﬁrst drawing N samples, { x (0) n } Nn =1 , from π 0 ( · ) = p ( · ) , noting that this can be done exactly because the perturbation model is known.
Revised: Our approach starts by ﬁrst drawing N samples, { x (0) n } Nn =1 , from π 0 ( · ) = p ( · ) , noting that this can be done exactly because the perturbation model is known.

Original: These samples can then be used to estimate P 1 using simple Monte Carlo,
Revised: These samples can then be used to estimate P 1 using simple Monte Carlo,

Original: [Equation]
Revised: [Equation]

Original: where x (0) n ∼ π0 ( · ) .
Revised: where x [Equation] [Equation] π

Original:  In other words, P 1 is the fraction of these samples whose property is greater than L 1 .
Revised: [Equation] [Equation] In other words, P 1 is the fraction of these samples whose property is greater than L 1 .

Original: Critically, by ensuring the value of L 1 is sufﬁciently small for { s ( x n ) ≥ L 1 } to be a common event, we can ensure ˆ P 1 is a reliable estimate for moderate numbers of samples N .
Revised: Critically, by ensuring the value of L 1 is sufﬁciently small for { s ( x n ) ≥ L 1 } to be a common event, we can ensure ˆ P 1 is a reliable estimate for moderate numbers of samples N .

Original: To estimate the other P k , we need to be able to draw samples from π k − 1 ( · ) .
Revised: To estimate the other P k , we need to be able to draw samples from π k − 1 ( · ) .

Original: For this we note that if { x ( k − 2) n } Nn =1 are distributed according to π k − 2 ( · ) , then the subset of these samples for which s ( x ( k − 2) n ) ≥ L k − 1 are distributed according to π k − 1 ( · ) .
Revised: For this we note that if { x ( k − 2) n } Nn =1 are distributed according to π k − 2 ( · ) , then the subset of these samples for which s ( x ( k − 2) n ) ≥ L k − 1 are distributed according to π k − 1 ( · ) .

Original: Furthermore, setting L k − 1 up to ensure this event is not rare means a signiﬁcant proportion of the samples will satisfy this property.
Revised: Furthermore, setting L k − 1 up to ensure this event is not rare means a signiﬁcant proportion of the samples will satisfy this property.

Original: To avoid our set of samples shrinking from one level to the next, it is necessary to carry out a rejuvenation step to convert this smaller set of starting samples to a full set of size N for the next level.
Revised: To avoid our set of samples shrinking from one level to the next, it is necessary to carry out a rejuvenation step to convert this smaller set of starting samples to a full set of size N for the next level.

Original: To do this, we ﬁrst carry out a uniform resampling with replacement from the set of samples satisfying s ( x ( k − 1) n ) ≥ L k to generate a new set of N samples which are distributed according to π k ( · ) , but with a large number of duplicated samples.
Revised: To do this, we ﬁrst carry out a uniform resampling with replacement from the set of samples satisfying s ( x ( k − 1) n ) ≥ L k to generate a new set of N samples which are distributed according to π k ( · ) , but with a large number of duplicated samples.

Original: Starting with these samples, we then successively apply
Revised: Starting with these samples, we then successively apply

Original: M Metropolis–Hastings (MH) transitions targeting π k ( · ) separately to each sample to produce a fresh new set of samples { x ( k ) n } N n =1 (see Appendix A for full details).
Revised: M Metropolis–Hastings (MH) transitions targeting π k ( · ) separately to each sample to produce a fresh new set of samples { x ( k ) n } Nn =1 (see Appendix A for full details).

Original: These samples can then in turn be used to form a Monte Carlo estimate for P k ,
Revised: These samples can then in turn be used to form a Monte Carlo estimate for P k ,

Original: [Equation]
Revised: [Equation]

Original: where 
Revised: where x ( k [Equation] 1) n

Original: [Equation]
Revised: [Equation]

Original:  along with providing the initializations for the next level.
Revised: 6) along with providing the initializations for the next level.

Original: Running more MH transitions decreases the correlations between the set of samples, improving the performance of the estimator.
Revised: Running more MH transitions decreases the correlations between the set of samples, improving the performance of the estimator.

Original: We have thus far omitted to discuss how the levels L k are set, other than asserting the need for the levels to be sufﬁciently close to allow reliable estimation of each P k .
Revised: We have thus far omitted to discuss how the levels L k are set, other than asserting the need for the levels to be sufﬁciently close to allow reliable estimation of each P k .

Original: Presuming that we are also free to choose the number of levels K , there is inevitably a trade-off between ensuring that each { s ( X ) ≥ L k } is not rare given { s ( X ) ≥ L k − 1 } , and keeping the number of levels small to reduce computational costs and avoid the build-up of errors.
Revised: Presuming that we are also free to choose the number of levels K , there is inevitably a trade-off between ensuring that each { s ( X ) ≥ L k } is not rare given { s ( X ) ≥ L k − 1 } , and keeping the number of levels small to reduce computational costs and avoid the build-up of errors.

Original: AMLS (Guyader et al., 2011) builds on the basic multi-level splitting process, providing an elegant way of controlling this trade-off by adaptively selecting the level to be the minimum of 0 and some quantile of the property under the current samples.
Revised: AMLS (Guyader et al., 2011) builds on the basic multi-level splitting process, providing an elegant way of controlling this trade-off by adaptively selecting the level to be the minimum of 0 and some quantile of the property under the current samples.

Original: The approach terminates when the level reaches zero, such that L K = 

Revised: The approach terminates when the level reaches zero, such that L K =

Original: and K is a dynamic parameter chosen implicitly by the adaptive process.
Revised: and K is a dynamic parameter chosen implicitly by the adaptive process.

Original: Choosing the ρ th quantile of the values of the property results in discarding a fraction (1 − ρ ) of the chains at each step of the algorithm.
Revised: Choosing the ρ th quantile of the values of the property results in discarding a fraction (1 − ρ ) of the chains at each step of the algorithm.

Original: This allows explicit control of the rarity of the events to keep them at a manageable level.
Revised: This allows explicit control of the rarity of the events to keep them at a manageable level.

Original: We note that if all the sample property values are distinct, then this approach gives P k = ρ, ∀ k < K .
Revised: We note that if all the sample property values are distinct, then this approach gives P k = ρ, ∀ k < K .

Original: To give intuition to this, we can think about splitting up log( I ) into chunks of size log( ρ ) .
Revised: To give intuition to this, we can think about splitting up log( I ) into chunks of size log( ρ ) .

Original: For any value of log( I ) , there is always a unique pair of values { K, P K } such that log( I ) =
Revised: For any value of log( I ) , there is always a unique pair of values { K, P K } such that log( I ) =

Original: K log( ρ ) + log( P K ) , K ≥ 0 and P K < ρ .
Revised: K log( ρ ) + log( P K ) , K ≥ 0 and P K < ρ .

Original: Therefore the problem of estimating
Revised: Therefore the problem of estimating

Original: I is equivalent to that of estimating K and P K .
Revised: I is equivalent to that of estimating K and P K .

Original: 5. T ERMINATION C RITERION
Revised: 5. T ERMINATION C RITERION

Original: The application of AMLS to our veriﬁcation problem presents a signiﬁcant complicating factor in that the true probability of our rare event might be exactly zero.
Revised: The application of AMLS to our veriﬁcation problem presents a signiﬁcant complicating factor in that the true probability of our rare event might be exactly zero.

Original: Whenever this is the case, the basic AMLS approach outlined in (Guyader et al., 2011) will never terminate as the quantile of the property will never rise above zero; the algorithm simply produces closer and closer intermediate levels as it waits for the event to occur.
Revised: Whenever this is the case, the basic AMLS approach outlined in (Guyader et al., 2011) will never terminate as the quantile of the property will never rise above zero; the algorithm simply produces closer and closer intermediate levels as it waits for the event to occur.

Original: To deal with this, we introduce a termination criterion based on the observation that AMLS’s running estimate for I monotonically decreases during running.
Revised: To deal with this, we introduce a termination criterion based on the observation that AMLS’s running estimate for I monotonically decreases during running.

Original: Namely, we introduce a threshold probability, P min , below which the estimates will be treated as being numerically zero.
Revised: Namely, we introduce a threshold probability, P min , below which the estimates will be treated as being numerically zero.

Original: We then terminate the algorithm if I < P min and return I = 0 , safe in the knowledge that even if the algorithm would eventually generate a ﬁnite estimate for I , this estimate is guaranteed to be less than P min .
Revised: We then terminate the algorithm if I < P min and return I = 0 , safe in the knowledge that even if the algorithm would eventually generate a ﬁnite estimate for I , this estimate is guaranteed to be less than P min .

Original: Putting everything together, gives the complete method as shown in Algorithm 1.
Revised: Putting everything together, gives the complete method as shown in Algorithm 1.

Original: See Appendix B for low-level implementation details.
Revised: See Appendix B for low-level implementation details.

Original:  E XPERIMENTS 6. E MULATION OF FORMAL VERIFICATION
Revised:  E XPERIMENTS 6. E MULATION OF FORMAL VERIFICATION

Original: In our ﬁrst experiment, we aim to test whether our statistical veriﬁcation framework is able to effectively emulate formal veriﬁcation approaches, while providing additional robustness information[Figure] for SAT properties.
Revised: In our ﬁrst experiment, we aim to test whether our robustness estimation framework is able to effectively emulate formal veriﬁcation approaches, while providing additional robustness information for SAT properties.

Original: In particular, we want to test whether it reliably identiﬁes properties as being
Revised: In particular, we want to test whether it reliably identiﬁes properties as being

Original: UNSAT , for which I = 0 , or SAT , for which I > 0 .
Revised: UNSAT , for which I = 0 , or SAT , for which I > 0 .

Original: We note that the method still provides a formal demonstration for SAT properties because having a non-zero estimate for I indicates that at least one counterexample has been found.
Revised: We note that the method still provides a formal demonstration for SAT properties because having a non-zero estimate for I indicates that at least one counterexample has been found.

Original: Critically, it further provides a measure for how robust SAT properties are, through its estimate for I .
Revised: Critically, it further provides a measure for how robust SAT properties are, through its estimate for I .

Original: We used the C OLLISION D ETECTION dataset introduced in the formal veriﬁcation literature by (Ehlers, 2017).
Revised: We used the C OLLISION D ETECTION dataset introduced in the formal veriﬁcation literature by (Ehlers, 2017).

Original: It consists of a neural network with six inputs that has been trained to classify two car trajectories as colliding or non-colliding.
Revised: It consists of a neural network with six inputs that has been trained to classify two car trajectories as colliding or non-colliding.

Original: The architecture has 40 linear nodes in the ﬁrst layer, followed by a layer of max pooling, a ReLU layer with 19 hidden units, and an output layer with 2 hidden units.
Revised: The architecture has 40 linear nodes in the ﬁrst layer, followed by a layer of max pooling, a ReLU layer with 19 hidden units, and an output layer with 2 hidden units.

Original: Along with the dataset, 500 properties are speciﬁed for veriﬁcation, of which 172 are SAT and 328 UNSAT .
Revised: Along with the dataset, 500 properties are speciﬁed for veriﬁcation, of which 172 are SAT and 328 UNSAT .

Original: This dataset was chosen because the model is small enough so that the properties can be formally veriﬁed.
Revised: This dataset was chosen because the model is small enough so that the properties can be formally veriﬁed.

Original: These formal veriﬁcation methods do not calculate the value of
Revised: These formal veriﬁcation methods do not calculate the value of

Original: I , but rather conﬁrm the existence of a counterexample for which s ( x ) > 0 .
Revised: I , but rather conﬁrm the existence of a counterexample for which s ( x ) > 0 .

Original: We ran our approach on all 500 properties, setting ρ = 0 .
Revised: We ran our approach on all 500 properties, setting ρ = 0 .

Original: 1 , N = 10 4 , M = 1000 (the choice of these hyperparameters will be justiﬁed in the next subsection), and using a uniform distribution over the input constraints as the perturbation model, along with a uniform random walk proposal.
Revised: 1 , N = 10 4 , M = 1000 (the choice of these hyperparameters will be justiﬁed in the next subsection), and using a uniform distribution over the input constraints as the perturbation model, along with a uniform random walk proposal.

Original: We compared our metric estimation approach against the naive Monte Carlo estimate using 10samples.
Revised: We compared our metric estimation approach against the naive Monte Carlo estimate using 10samples.

Original: The generated estimates of I for all SAT properties are shown in Figure 1a.
Revised: The generated estimates of I for all SAT properties are shown in Figure 1a.

Original: Both our approach and the naive MC baseline correctly identiﬁed all of the UNSAT properties by estimating I as exactly zero.
Revised: Both our approach and the naive MC baseline correctly identiﬁed all of the UNSAT properties by estimating I as exactly zero.

Original: However, despite using substantially more samples, naive MC failed to ﬁnd a counterexample for 8 of the rarest SAT properties, thereby identifying them as UNSAT , whereas our approach found a counterexample for all the SAT properties.
Revised: However, despite using substantially more samples, naive MC failed to ﬁnd a counterexample for 8 of the rarest SAT properties, thereby identifying them as UNSAT , whereas our approach found a counterexample for all the SAT properties.

Original: As shown in Figure 1a, the variances in the estimates for I of our approach were also very low and matched the unbiased
Revised: As shown in Figure 1a, the variances in the estimates for I of our approach were also very low and matched the unbiased

Original: MC baseline estimates for the more commonly violated properties, for which the latter approach still gives reliable, albeit less efﬁcient, estimates.
Revised: MC baseline estimates for the more commonly violated properties, for which the latter approach still gives reliable, albeit less efﬁcient, estimates.

Original: Along with the improved ability to predict rare events, our approach was also signiﬁcantly faster than naive MC throughout, with a speed up of several orders of magnitude for properties where the event is not rare—a single run with naive MC took about 3 minutes, whereas a typical run of ours took around 3 seconds.
Revised: Along with the improved ability to predict rare events, our approach was also signiﬁcantly faster than naive MC throughout, with a speed up of several orders of magnitude for properties where the event is not rare—a single run with naive MC took about 3 minutes, whereas a typical run of ours took around 3 seconds.

Original: 6. S ENSITIVITY TO P ARAMETER S ETTINGS
Revised: 6. S ENSITIVITY TO P ARAMETER S ETTINGS

Original: As demonstrated by Br´ehier et al.
Revised: As demonstrated by Br´ehier et al.

Original: (2015), AMLS is unbiased under the assumption that perfect sampling from the targets, { π k } K − 1 k =1 , is possible, and that the cumulative distribution function of s ( X ) is continuous.
Revised: (2015), AMLS is unbiased under the assumption that perfect sampling from the targets, { π k } K − 1 k =1 , is possible, and that the cumulative distribution function of s ( X ) is continuous.

Original: In practice, ﬁnite mixing rates of the Markov chains and the dependence between the initialization points for each target means that sampling is less than perfect, but improves with larger values of M and N .
Revised: In practice, ﬁnite mixing rates of the Markov chains and the dependence between the initialization points for each target means that sampling is less than perfect, but improves with larger values of M and N .

Original: The variance, on the other hand, theoretically strictly decreases with larger values of N and ρ (Br´ehier et al., 2015).
Revised: The variance, on the other hand, theoretically strictly decreases with larger values of N and ρ (Br´ehier et al., 2015).

Original: [Figure]

Original: In practice, we found that while larger values of M and N were always beneﬁcial, setting ρ too high introduced biases into the estimate, with ρ = 0 .
Revised: In practice, we found that while larger values of M and N were always beneﬁcial, setting ρ too high introduced biases into the estimate, with ρ = 0 .

Original: 1 empirically providing a good trade-off between bias and variance.
Revised: 1 empirically providing a good trade-off between bias and variance.

Original: Furthermore, this provides faster run times than large values of ρ , noting that the smaller values of ρ lead to larger gaps in the levels.
Revised: Furthermore, this provides faster run times than large values of ρ , noting that the smaller values of ρ lead to larger gaps in the levels.

Original: To investigate the effect of the parameters more formally, we further ran AMLS on the SAT properties of C OLLISION D ETECTION , varying ρ ∈ { 0 . 1 , 0 .
Revised: To investigate the effect of the parameters more formally, we further ran AMLS on the SAT properties of C OLLISION D ETECTION , varying ρ ∈ { 0 . 1 , 0 .

Original: 25 , 0 . 5 } , N ∈ { 10 3 , 10 4 , 10 5 } and
Revised: 25 , 0 . 5 } , N ∈ { 10 3 , 10 4 , 10 5 } and

Original:  M ∈ { 100 , 250 , 1000 } , again comparing to thenaive MC estimate for 10 10 samples.
Revised: [Figure] M ∈ { 100 , 250 , 1000 } , again comparing to the MC estimate for 10 10 samples.

Original: We found that the value of N did not make a discernible difference in this range regardless of the values for ρ and M , and thus all presented results correspond to setting N = 10 4 .
Revised: We found that the value of N did not make a discernible difference in this range regardless of the values for ρ and M , and thus all presented results correspond to setting N = 10 4 .

Original: As shown in Figure 1b, we found that the setting of ρ made a noticeable difference to the estimates for the relatively rarer events.
Revised: As shown in Figure 1b, we found that the setting of ρ made a noticeable difference to the estimates for the relatively rarer events.

Original: All the same, these differences were small relative to the differences between properties.
Revised: All the same, these differences were small relative to the differences between properties.

Original: As shown in Figure 1c, the value of M made little difference when ρ = 0 .
Revised: As shown in Figure 1c, the value of M made little difference when ρ = 0 .

Original: Interesting though, we found that the value of M was important for different values of ρ , as shown in Appendix C.1, with larger values of M giving better results as expected.
Revised: Interesting though, we found that the value of M was important for different values of ρ , as shown in Appendix C.1, with larger values of M giving better results as expected.

Original: 6. C ONVERGENCE WITH HIGHER - DIMENSIONAL INPUTS AND LARGER NETWORKS
Revised: 6. C ONVERGENCE WITH HIGHER - DIMENSIONAL INPUTS AND LARGER NETWORKS

Original: To validate the algorithm on a higher-dimensional problem, we ﬁrst tested adversarial properties on the MNIST and CIFAR –10 datasets using a dense ReLU network with two hidden-layer of size 256 .
Revised: To validate the algorithm on a higher-dimensional problem, we ﬁrst tested adversarial properties on the MNIST and CIFAR –10 datasets using a dense ReLU network with two hidden-layer of size 256 .

Original: An l ∞ -norm ball perturbation around the data point with width (cid:15) was used as the uniform input model, with (cid:15)
Revised: An l ∞ -norm ball perturbation around the data point with width (cid:15) was used as the uniform input model, with (cid:15)

Original: =
Revised: =

Original: 1 representing an l ∞ -ball ﬁlling the entire space (the pixels are scaled to [0 , 1] ), together with a uniform random walk MH proposal.
Revised: 1 representing an l ∞ -ball ﬁlling the entire space (the pixels are scaled to [0 , 1] ), together with a uniform random walk MH proposal.

Original: After training the classiﬁers, multilevel splitting was run on ten samples from the test set at multiple values of (cid:15) , with N = 10000 and ρ = 0 .
Revised: After training the classiﬁers, multilevel splitting was run on ten samples from the test set at multiple values of (cid:15) , with N = 10000 and ρ = 0 .

Original: 1 , and
Revised: 1 , and

Original: M ∈ { 100 , 250 , 1000 } for MNIST and M ∈ { 100 , 250 , 500 , 1000 , 2000 } for CIFAR –10.
Revised: M ∈ { 100 , 250 , 1000 } for MNIST and M ∈ { 100 , 250 , 500 , 1000 , 2000 } for CIFAR –10.

Original: The results for naive MC were also evaluated using 5 × 10 9 samples—less than the previous experiment as the larger network made estimation more expensive—in the cases where the event was not too rare.
Revised: The results for naive MC were also evaluated using 5 × 10 9 samples—less than the previous experiment as the larger network made estimation more expensive—in the cases where the event was not too rare.

Original: This took around twenty minutes per naive MC estimate, versus a few minutes for each AMLS estimate.
Revised: This took around twenty minutes per naive MC estimate, versus a few minutes for each AMLS estimate.

Original: As the results were similar across datapoints, we present the result for a single example in the top two rows of Figure 2.
Revised: As the results were similar across datapoints, we present the result for a single example in the top two rows of Figure 2.

Original: As desired, a smooth curve is traced out as (cid:15) decreases, for which the event E becomes rarer.
Revised: As desired, a smooth curve is traced out as (cid:15) decreases, for which the event E becomes rarer.

Original: For MNIST , acceptable accuracy is obtained for M = 250 and high accuracy results for M = 1000 .
Revised: For MNIST , acceptable accuracy is obtained for M = 250 and high accuracy results for M = 1000 .

Original: For CIFAR –10, which has about four times the input dimension of MNIST , larger values of M were required to achieve comparable accuracy.
Revised: For CIFAR –10, which has about four times the input dimension of MNIST , larger values of M were required to achieve comparable accuracy.

Original: The magnitude of (cid:15) required to give a certain value of log( I ) is smaller for CIFAR –10 than MNIST , reﬂecting that adversarial examples for the former are typically more perceptually similar to the datapoint.
Revised: The magnitude of (cid:15) required to give a certain value of log( I ) is smaller for CIFAR –10 than MNIST , reﬂecting that adversarial examples for the former are typically more perceptually similar to the datapoint.

Original: To demonstrate that our approach can be employed on large networks, we tested adversarial properties on the CIFAR –100 dataset and a much larger DenseNet architecture (Huang et al., 2017a), with depth and growth-rate 40 (approximately 2 × 10 6 parameters).
Revised: To demonstrate that our approach can be employed on large networks, we tested adversarial properties on the CIFAR –100 dataset and a much larger DenseNet architecture (Huang et al., 2017a), with depth and growth-rate 40 (approximately 2 × 10 6 parameters).

Original: Due to the larger model size, we set N = 300 , the largest minibatch that could be held in memory (a larger N could be used by looping over minibatches).
Revised: Due to the larger model size, we set N = 300 , the largest minibatch that could be held in memory (a larger N could be used by looping over minibatches).

Original: The naive Monte Carlo estimates used 5 × 10 6 samples for about an hour of computation time per estimate, compared to between ﬁve to ﬁfteen minutes for each AMLS estimate.
Revised: The naive Monte Carlo estimates used 5 × 10 6 samples for about an hour of computation time per estimate, compared to between ﬁve to ﬁfteen minutes for each AMLS estimate.

Original: The results are presented in the bottom row of Figure 2, showing that our algorithm agrees with the naive Monte Carlo estimate. 
Revised: The results are presented in the bottom row of Figure 2, showing that our algorithm agrees with the naive Monte Carlo estimate. [Figure]

Original: 6. R OBUSTNESS OF PROVABLE DEFENSES DURING TRAINING
Revised: 6. R OBUSTNESS OF PROVABLE DEFENSES DURING TRAINING

Original: We now examine how our robustness metric varies for a ReLU network as that network is trained to be more robust against norm bounded perturbations to the inputs using the method of Wong & Kolter (2018).
Revised: We now examine how our robustness metric varies for a ReLU network as that network is trained to be more robust against norm bounded perturbations to the inputs using the method of Wong & Kolter (2018).

Original: Roughly speaking, their method works by approximating the set of outputs resulting from perturbations to an input with a convex outer bound, and minimizing the worst case loss over this set.
Revised: Roughly speaking, their method works by approximating the set of outputs resulting from perturbations to an input with a convex outer bound, and minimizing the worst case loss over this set.

Original: The motivation for this experiment is twofold.
Revised: The motivation for this experiment is twofold.

Original: Firstly, this training provides a series of networks with ostensibly increasing robustness, allowing us to check if our approach produces robustness estimates consistent with this improvement.
Revised: Firstly, this training provides a series of networks with ostensibly increasing robustness, allowing us to check if our approach produces robustness estimates consistent with this improvement.

Original: Secondly, it allows us to investigate whether the training to improve robustness for one type of adversarial attack helps to protect against others.
Revised: Secondly, it allows us to investigate whether the training to improve robustness for one type of adversarial attack helps to protect against others.

Original: Speciﬁcally, whether training for small perturbation sizes improves robustness to larger perturbations.
Revised: Speciﬁcally, whether training for small perturbation sizes improves robustness to larger perturbations.

Original: We train a CNN model on MNIST for 100 epochs with the standard cross-entropy loss, then train the network for a further 100 epochs using the robust loss of Wong & Kolter (2018), saving a snapshot of the model at each epoch. 
Revised: We train a CNN model on MNIST for 100 epochs with the standard cross-entropy loss, then train the network for a further 100 epochs using the robust loss of Wong & Kolter (2018), saving a snapshot of the model at each epoch. The architecture is the same as in (Wong & Kolter, 2018), containing [Figure] . 3 } , and (cid:15)

Original: The architecture is the same as in (Wong & Kolter, 2018), containing two strided convolutional layers with 16 and 32 channels, followed by two fully connected layers with 100 and 10 hidden units, and ReLU activations throughout.
Revised: = 0 . 1 before epoch 32 with the method of Wong & Kolter (2018). Our metric, however, suffers no such memory issues. two strided convolutional layers with 16 and 32 channels, followed by two fully connected layers with 100 and 10 hidden units, and ReLU activations throughout.

Original: The robustiﬁcation phase trains the classiﬁer to be robust in an l ∞ (cid:15) -ball around the inputs, where (cid:15) is annealed from 0 .
Revised: The robustiﬁcation phase trains the classiﬁer to be robust in an l ∞ (cid:15) -ball around the inputs, where (cid:15) is annealed from 0 .

Original: 01 to 0 .
Revised: 01 to 0 .

Original: 1 over the ﬁrst 50 epochs.
Revised: 1 over the ﬁrst 50 epochs.

Original: At a number of epochs during the robust training, we calculate our robustness metric with (cid:15) ∈ { 0 . 1 , 0 . 2 , 0 . 3 } on 50 samples from the test set.
Revised: At a number of epochs during the robust training, we calculate our robustness metric with (cid:15) ∈ { 0 . 1 , 0 . 2 , 0 . 3 } on 50 samples from the test set.

Original: The results are summarized in Figure 3a with additional per-sample results in Appendix C.2.
Revised: The results are summarized in Figure 3a with additional per-sample results in Appendix C.2.

Original: We see that our approach is able to capture variations in the robustnesses of the network.
Revised: We see that our approach is able to capture variations in the robustnesses of the network.

Original: As the method of Wong & Kolter (2018) returns the maximum value of the property for each sample over a convex outer bound on the perturbations, it is able to produce certiﬁcates-of-robustness for some datapoints.
Revised: As the method of Wong & Kolter (2018) returns the maximum value of the property for each sample over a convex outer bound on the perturbations, it is able to produce certiﬁcates-of-robustness for some datapoints.

Original: If the result returned is less than 

Revised: If the result returned is less than

Original: then no adversarial examples exist in an l ∞ ball of radius (cid:15) around that datapoint.
Revised: then no adversarial examples exist in an l ∞ ball of radius (cid:15) around that datapoint.

Original: If the result returned is greater than 0 , then the datapoint may or may not be robust in that l ∞ ball, due to fact that it optimizes over an outer bound.
Revised: If the result returned is greater than 0 , then the datapoint may or may not be robust in that l ∞ ball, due to fact that it optimizes over an outer bound.

Original: Though we emphasize that the core aim of our approach is in providing richer information for SAT properties, this provides an opportunity to see how well it performs at establishing UNSAT properties relative to a more classical approach.
Revised: Though we emphasize that the core aim of our approach is in providing richer information for SAT properties, this provides an opportunity to see how well it performs at establishing UNSAT properties relative to a more classical approach.

Original: To this end, we compared the fraction of the 50 samples from the test set that are veriﬁed by the method of Wong & Kolter (2018), to the fraction that have a [Figure] 0 . 3 } , and (cid:15) = 0 . 1 before epoch 32 with the method of Wong & Kolter (2018).
Revised: To this end, we compared the fraction of the 50 samples from the test set that are veriﬁed by the method of Wong & Kolter (2018), to the fraction that have a negligible volume of adversarial examples, I = P min , in their l ∞ (cid:15) -ball neighbourhood.

Original: Our metric, however, suffers no such memory issues.

Original: negligible volume of adversarial examples, I = P min , in their l ∞ (cid:15) -ball neighbourhood.

Original: The results are presented in Figure 3b.
Revised: The results are presented in Figure 3b.

Original: Our method forms an upper bound on the fraction of robust samples, which can be made arbitrarily tighter by taking P min → 0 .
Revised: Our method forms an upper bound on the fraction of robust samples, which can be made arbitrarily tighter by taking P min → 0 .

Original: Wong & Kolter (2018), on the other hand, forms a lower bound on the fraction of robust samples, where the tightness of the bound depends on the tightness of the convex outer bound, which is unknown and cannot be controlled.
Revised: Wong & Kolter (2018), on the other hand, forms a lower bound on the fraction of robust samples, where the tightness of the bound depends on the tightness of the convex outer bound, which is unknown and cannot be controlled.

Original: Though the true value must lie somewhere between the two bounds, our bound still holds physical meaning it its own right in a way that Wong & Kolter (2018) does not: it is the proportion of samples for which the prevalence of violations is less than an a given acceptable threshold P min .
Revised: Though the true value must lie somewhere between the two bounds, our bound still holds physical meaning it its own right in a way that Wong & Kolter (2018) does not: it is the proportion of samples for which the prevalence of violations is less than an a given acceptable threshold P min .

Original: This experiment also highlights an important shortcoming of Wong & Kolter (2018).
Revised: This experiment also highlights an important shortcoming of Wong & Kolter (2018).

Original: The memory usage of their procedure depends on how many ReLU activations cross their threshold over perturbations.
Revised: The memory usage of their procedure depends on how many ReLU activations cross their threshold over perturbations.

Original: This is high during initial training for (cid:15) = 0 .
Revised: This is high during initial training for (cid:15) = 0 .

Original: 1 and indeed the reason why the training procedure starts from (cid:15) = 0 .
Revised: 1 and indeed the reason why the training procedure starts from (cid:15) = 0 .

Original: 01 and gradually anneals to (cid:15) = 0 .
Revised: 01 and gradually anneals to (cid:15) = 0 .

Original: The result is that it is infeasible (the GPU memory is exhausted)—even for this relatively small model—to calculate the maximum value of the property on the convex outer bound for (cid:15) ∈ { 0 . 2 , 0 . 3 } at all epochs, and (cid:15) = 0 .
Revised: The result is that it is infeasible (the GPU memory is exhausted)—even for this relatively small model—to calculate the maximum value of the property on the convex outer bound for (cid:15) ∈ { 0 . 2 , 0 . 3 } at all epochs, and (cid:15) = 0 .

Original: 1 for epochs before 32 .
Revised: 1 for epochs before 32 .

Original: Even in this restricted setting where our metric has been reduced to a binary one, it appears to be more informative than that of Wong & Kolter (2018) for this reason.
Revised: Even in this restricted setting where our metric has been reduced to a binary one, it appears to be more informative than that of Wong & Kolter (2018) for this reason.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have introduced a new measure for the intrinsic robustness of a neural network, and have validated its utility on several datasets from the formal veriﬁcation and deep learning literatures.
Revised: We have introduced a new measure for the intrinsic robustness of a neural network, and have validated its utility on several datasets from the formal veriﬁcation and deep learning literatures.

Original: Our approach was able to exactly emulate formal veriﬁcation approaches for satisﬁable properties and provide high conﬁdence, accurate predictions for properties which were not.
Revised: Our approach was able to exactly emulate formal veriﬁcation approaches for satisﬁable properties and provide high conﬁdence, accurate predictions for properties which were not.

Original: The two key advantages it provides over previous approaches are: a) providing an explicit and intuitive measure for how robust networks are to satisﬁable properties; and b) providing improved scaling over classical approaches for identifying unsatisﬁable properties.
Revised: The two key advantages it provides over previous approaches are: a) providing an explicit and intuitive measure for how robust networks are to satisﬁable properties; and b) providing improved scaling over classical approaches for identifying unsatisﬁable properties.

Revised: Despite providing a more informative measure of how robust a neural network is, our approach may not be appropriate in all circumstances.

Original: Going forward, an interesting avenue that could be exploited further is  the fact that our approach has the highly desirable property of producing a diverse set of counterexamples as a by-product.
Revised: In situations where there is an explicit and effective adversary, instead of inputs being generated by chance, we may care more about how far away the single closest counterexample is to the input, rather than the general prevalence of counterexamples. Here our method may fail to ﬁnd counterexamples because they reside on a subset with probability less than P min ; the counterexamples may even reside on a subset of the input space with measure zero with respect to the input distribution. On the other hand, there are many practical scenarios, such as those discussed in the introduction, where either it is unrealistic for there to be no counterexamples close to the input, the network (or input space) is too large to realistically permit formal veriﬁcation, or where potential counterexamples are generated by chance rather than by an adversary. We believe that for these scenarios our approach offers signiﬁcant advantages to formal veriﬁcation approaches.

Original: In many cases these are likely to be of notable interest in themselves, for example, potentially paving the way for new approaches to training robust networks and protecting against adversarial attacks.
Revised: Going forward, one way the efﬁciency of our approach could be improved further is by using a more efﬁcient base MCMC kernel in our AMLS estimator, that is replace line 12 in Algorithm 1 with a more efﬁcient base inference scheme. The current MH scheme was chosen on the basis of simplicity and the fact it already gave effective empirical performance. However, using more advanced inference approaches, such as gradient-based approaches like Langevin Monte Carlo (LMC) (Rossky et al., 1978) and Hamiltonian Monte Carlo (Neal, 2011), could provide signiﬁcant speedups by improving the mixing of the Markov chains, thereby reducing the number of required MCMC transitions.

Original: A PPENDIX A M ETROPOLIS –H ASTINGS
Revised: A CKNOWLEDGMENTS

Revised: We gratefully acknowledge Sebastian Nowozin for suggesting to us to apply multilevel splitting to the problem of estimating neural network robustness. We also thank Rudy Bunel for his help with the C OLLISION D ETECTION dataset, and Leonard Berrada for supplying a pretrained DenseNet model.

Revised: SW gratefully acknowledges support from the EPSRC AIMS CDT through grant EP/L015987/2.

Revised: TR and YWT are supported in part by the European Research Council under the European Unions

Revised: Seventh Framework Programme (FP7/20072013) / ERC grant agreement no. 617071.

Revised: TR further acknowledges support of the ERC StG IDIU.

Original:  Metropolis–Hastings (MH) is an MCMC method that allows for sampling when one only has access to an unnormalized version of the target distribution (Gilks et al., 1995).
Revised: MPK is supported by EPSRC grants EP/P020658/ and TU/B/000048. A PPENDIX A M ETROPOLIS –H ASTINGS Metropolis–Hastings (MH) is an MCMC method that allows for sampling when one only has access to an unnormalized version of the target distribution (Gilks et al., 1995).

Original: At a high-level, one attempts iteratively proposes local moves from the current location of a sampler and then accepts or rejects this move based on the unnormalized density.
Revised: At a high-level, one attempts iteratively proposes local moves from the current location of a sampler and then accepts or rejects this move based on the unnormalized density.

Original: Each iteration of this process is known as a
Revised: Each iteration of this process is known as a

Original: MH transition.
Revised: MH transition.

Original: The unnormalized targets distributions of interest for our problem are γ k ( x ) where
Revised: The unnormalized targets distributions of interest for our problem are γ k ( x ) where

Original: [Equation]
Revised: [Equation]

Original: A MH transition now consists of proposing a new sample using a proposal x (cid:48) ∼ g ( x (cid:48)
Revised: A MH transition now consists of proposing a new sample using a proposal x (cid:48) ∼ g ( x (cid:48)

Original: | x ) , where x indicates the current state of the sampler and x (cid:48)
Revised: | x ) , where x indicates the current state of the sampler and x (cid:48)

Original: the proposed state, calculating an acceptance probability,
Revised: the proposed state, calculating an acceptance probability,

Original: [Equation]
Revised: [Equation]

Original: and accepting the new sample with probability A k ( x (cid:48) | x ) , returning the old sample if the new one is rejected.
Revised: and accepting the new sample with probability A k ( x (cid:48) | x ) , returning the old sample if the new one is rejected.

Original: The proposal, g ( x (cid:48) | x ) , is a conditional distribution, such as a normal distribution centred at x with ﬁxed covariance matrix.
Revised: The proposal, g ( x (cid:48) | x ) , is a conditional distribution, such as a normal distribution centred at x with ﬁxed covariance matrix.

Original: Successive applications of this transition process generates samples which converge in distribution to the target π k ( x ) and whose correlation with the starting sample diminishes to zero.
Revised: Successive applications of this transition process generates samples which converge in distribution to the target π k ( x ) and whose correlation with the starting sample diminishes to zero.

Original: In our approach, these MH steps are applied independently to each sample in the set, while the only samples used for the AMLS algorithm are the ﬁnal samples produced from the resulting Markov chains.
Revised: In our approach, these MH steps are applied independently to each sample in the set, while the only samples used for the AMLS algorithm are the ﬁnal samples produced from the resulting Markov chains.

Original: B I MPLEMENTATION D ETAILS
Revised: B I MPLEMENTATION D ETAILS

Original: Algorithm 1 has computational cost O ( NMK ) , where the number of levels K will depend on the rareness of the event, with more computation required for rarer ones.
Revised: Algorithm 1 has computational cost O ( NMK ) , where the number of levels K will depend on the rareness of the event, with more computation required for rarer ones.

Original: Parallelization over N is possible provided that the batches ﬁt into memory, whereas the loops over M and K must be performed sequentially.
Revised: Parallelization over N is possible provided that the batches ﬁt into memory, whereas the loops over M and K must be performed sequentially.

Original: One additional change we make from the approach outlined by Guyader et al.
Revised: One additional change we make from the approach outlined by Guyader et al.

Original: (2011) is that we perform MH updates on all chains in Lines 12, rather than only those that were previously killed off.
Revised: (2011) is that we perform MH updates on all chains in Lines 12, rather than only those that were previously killed off.

Original: This helps reduce the build up of correlations over multiple levels, improving performance.
Revised: This helps reduce the build up of correlations over multiple levels, improving performance.

Original: Another is that we used an adaptive scheme for g ( x (cid:48) | x ) to aid efﬁciency.
Revised: Another is that we used an adaptive scheme for g ( x (cid:48) | x ) to aid efﬁciency.

Original: Speciﬁcally, our proposal takes the form of a random walk, the radius of which, (cid:15) (cid:48) , is adapted to keep the acceptance ratio roughly around 0 .
Revised: Speciﬁcally, our proposal takes the form of a random walk, the radius of which, (cid:15) (cid:48) , is adapted to keep the acceptance ratio roughly around 0 .

Original: 234 (see Roberts et al.
Revised: 234 (see Roberts et al.

Original: Each chain has a separate acceptance ratio that is average across MH steps, and after M MH steps, for those chains whose acceptance ratio is below 0.234 it is halved, and conversely for those above 0.234, multiplied by 1.02.
Revised: Each chain has a separate acceptance ratio that is average across MH steps, and after M MH steps, for those chains whose acceptance ratio is below 0.234 it is halved, and conversely for those above 0.234, multiplied by 1.02.

Original: C A DDITIONAL RESULTS C. V ARYING M FOR FIXED ρ ON C OLLISION D ETECTION
Revised: C A DDITIONAL RESULTS C. V ARYING M FOR FIXED ρ ON C OLLISION D ETECTION

Original: Whereas the exact value of M within the range considered proved to not be especially important when ρ = 0 .
Revised: Whereas the exact value of M within the range considered proved to not be especially important when ρ = 0 .

Original: 1 , it transpires to have a large impact in the quality of the results for larger values of ρ as shown in Figure 4.
Revised: 1 , it transpires to have a large impact in the quality of the results for larger values of ρ as shown in Figure 4.

Original: C. P ER - SAMPLE ROBUSTNESS MEASURE DURING ROBUST TRAINING
Revised: C. P ER - SAMPLE ROBUSTNESS MEASURE DURING ROBUST TRAINING

Original: Figure 5 illustrates the diverse forms that the per-sample robustness measure can take on thedatapoints averaged over in Experiment § 5.3.
Revised: Figure 5 illustrates the diverse forms that the per-sample robustness measure can take on thedatapoints averaged over in Experiment § 5.3.

Original: We see that different datapoints have quite varying initial levels of robustness, and that the training helps with some points more than others.
Revised: We see that different datapoints have quite varying initial levels of robustness, and that the training helps with some points more than others.

Original: In one case, the datapoint was still not robust add the end of training for the target perturbation size (cid:15) = 0 .
Revised: In one case, the datapoint was still not robust add the end of training for the target perturbation size (cid:15) = 0 .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]