Original: F AST G EOMETRIC P ROJECTIONS FOR L OCAL R OBUSTNESS C ERTIFICATION
Revised: F AST G EOMETRIC P ROJECTIONS FOR L OCAL R OBUSTNESS C ERTIFICATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Local robustness ensures that a model classiﬁes all inputs within an (cid:96) p -ball consistently, which precludes various forms of adversarial inputs.
Revised: Local robustness ensures that a model classiﬁes all inputs within an (cid:96) p -ball consistently, which precludes various forms of adversarial inputs.

Original: In this paper, we present a fast procedure for checking local robustness in feed-forward neural networks with piecewise-linear activation functions.
Revised: In this paper, we present a fast procedure for checking local robustness in feed-forward neural networks with piecewise-linear activation functions.

Original: Such networks partition the input space into a set of convex polyhedral regions in which the network’s behavior is linear; hence, a systematic search for decision boundaries within the regions around a given input is sufﬁcient for assessing robustness.
Revised: Such networks partition the input space into a set of convex polyhedral regions in which the network’s behavior is linear; hence, a systematic search for decision boundaries within the regions around a given input is sufﬁcient for assessing robustness.

Original: Crucially, we show how the regions around a point can be analyzed using simple geometric projections, thus admitting an efﬁcient, highly-parallel GPU implementation that excels particularly for the (cid:96) 2 norm, where previous work has been less effective.
Revised: Crucially, we show how the regions around a point can be analyzed using simple geometric projections, thus admitting an efﬁcient, highly-parallel GPU implementation that excels particularly for the (cid:96) 2 norm, where previous work has been less effective.

Original: Empirically we ﬁnd this approach to be far more precise than many approximate veriﬁcation approaches, while at the same time performing multiple orders of magnitude faster than complete veriﬁers, and scaling to much deeper networks.
Revised: Empirically we ﬁnd this approach to be far more precise than many approximate veriﬁcation approaches, while at the same time performing multiple orders of magnitude faster than complete veriﬁers, and scaling to much deeper networks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: We consider the problem of verifying the local robustness of piecewise-linear neural networks for a given (cid:96) p bound.
Revised: We consider the problem of verifying the local robustness of piecewise-linear neural networks for a given (cid:96) p bound.

Original: Precisely, given a point, x , network, F , and norm bound, (cid:15) , this entails determining whether Equation 1 holds.
Revised: Precisely, given a point, x , network, F , and norm bound, (cid:15) , this entails determining whether Equation 1 holds.

Original: [Equation]
Revised: [Equation]

Original: This problem carries practical signiﬁcance, as such networks have been extensively shown to be vulnerable to adversarial examples (Papernot et al., 2016; Szegedy et al., 2014), wherein small-norm perturbations are chosen to cause arbitrary misclassiﬁcations.
Revised: This problem carries practical signiﬁcance, as such networks have been extensively shown to be vulnerable to adversarial examples (Papernot et al., 2016; Szegedy et al., 2014), wherein small-norm perturbations are chosen to cause arbitrary misclassiﬁcations.

Original: Numerous solutions have been proposed to address variants of this problem.
Revised: Numerous solutions have been proposed to address variants of this problem.

Original: These can be roughly categorized into three groups: learning rules that aim for robustness on known training data (Croce et al., 2019; Madry et al., 2018; Wong & Kolter, 2018;
Revised: These can be roughly categorized into three groups: learning rules that aim for robustness on known training data (Croce et al., 2019; Madry et al., 2018; Wong & Kolter, 2018;

Original: Zhang et al., 2019; Xiao et al., 2019), post-processing methods that provide stochastic guarantees at inference time (Cohen et al., 2019; Lecuyer et al., 2018), and network veriﬁcation (Balunovic et al., 2019; Cheng et al., 2017; Dutta et al., 2018; Ehlers, 2017; Fischetti & Jo, 2018; Gowal et al., 2018;
Revised: Zhang et al., 2019; Xiao et al., 2019), post-processing methods that provide stochastic guarantees at inference time (Cohen et al., 2019; Lecuyer et al., 2018), and network veriﬁcation (Balunovic et al., 2019; Cheng et al., 2017; Dutta et al., 2018; Ehlers, 2017; Fischetti & Jo, 2018; Gowal et al., 2018;

Original: Jordan et al., 2019; Katz et al., 2017; 2019; Singh et al., 2019b; Tjeng & Tedrake, 2017; Wang et al., 2018; Weng et al., 2018).
Revised: Jordan et al., 2019; Katz et al., 2017; 2019; Singh et al., 2019b; Tjeng & Tedrake, 2017; Wang et al., 2018; Weng et al., 2018).

Original: We focus on the problem of network veriﬁcation—for a given model and input, determining if
Revised: We focus on the problem of network veriﬁcation—for a given model and input, determining if

Original: Equation 1 holds—particularly for the (cid:96) 2 norm.
Revised: Equation 1 holds—particularly for the (cid:96) 2 norm.

Original: Historically, the literature has primarily concentrated on the (cid:96) ∞ norm, with relatively little work on the (cid:96) 2 norm; indeed, many of the best-scaling veriﬁcation tools do not even support veriﬁcation with respect to the (cid:96) 2 norm.
Revised: Historically, the literature has primarily concentrated on the (cid:96) ∞ norm, with relatively little work on the (cid:96) 2 norm; indeed, many of the best-scaling veriﬁcation tools do not even support veriﬁcation with respect to the (cid:96) 2 norm.

Original: Nonetheless, the (cid:96) 2 norm remains important to consider for “imperceptible” adversarial examples (Rony et al., 2019).
Revised: Nonetheless, the (cid:96) 2 norm remains important to consider for “imperceptible” adversarial examples (Rony et al., 2019).

Original: Furthermore, compared to the (cid:96) ∞ norm, efﬁcient veriﬁcation for the (cid:96) 2 norm presents a particular challenge, as constraint-solving (commonly used in veriﬁcation tools) in Euclidean space requires a non-linear objective function, and cannot make as effective use of interval-bound propagation.
Revised: Furthermore, compared to the (cid:96) ∞ norm, efﬁcient veriﬁcation for the (cid:96) 2 norm presents a particular challenge, as constraint-solving (commonly used in veriﬁcation tools) in Euclidean space requires a non-linear objective function, and cannot make as effective use of interval-bound propagation.

Original: Existing work on verifying local robustness for the (cid:96) 2 norm falls into two primary categories: (1) expensive, but exact decision procedures, e.g., GeoCert (Jordan et al., 2019) and MIP (Tjeng & Tedrake, 2017), or (2) fast, but approximate techniques, e.g., FastLin/CROWN (Weng et al., 2018;
Revised: Existing work on verifying local robustness for the (cid:96) 2 norm falls into two primary categories: (1) expensive, but exact decision procedures, e.g., GeoCert (Jordan et al., 2019) and MIP (Tjeng & Tedrake, 2017), or (2) fast, but approximate techniques, e.g., FastLin/CROWN (Weng et al., 2018;

Original: Zhang et al., 2018).
Revised: Zhang et al., 2018).

Original: While approximate veriﬁcation methods have shown promise in scaling to larger networks, they may introduce an additional penalty to robust accuracy by ﬂagging non-adversarial points, thus limiting their application in practice.
Revised: While approximate veriﬁcation methods have shown promise in scaling to larger networks, they may introduce an additional penalty to robust accuracy by ﬂagging non-adversarial points, thus limiting their application in practice.

Original: Exact methods impose no such penalty, but as they rely on expensive constraint-solving techniques, they often do not scale well to even networks with a few hundred neurons.
Revised: Exact methods impose no such penalty, but as they rely on expensive constraint-solving techniques, they often do not scale well to even networks with a few hundred neurons.

Original: In this paper, we focus on bridging the gap between these two approaches.
Revised: In this paper, we focus on bridging the gap between these two approaches.

Original: In particular, we present a veriﬁcation technique for Equation 1 that neither relies on expensive constraint solving nor conservative over-approximation of the decision boundaries.
Revised: In particular, we present a veriﬁcation technique for Equation 1 that neither relies on expensive constraint solving nor conservative over-approximation of the decision boundaries.

Original: Our algorithm (Section 2) leverages simple projections, rather than constraint solving, to exhaustively search the model’s decision boundaries around a point.
Revised: Our algorithm (Section 2) leverages simple projections, rather than constraint solving, to exhaustively search the model’s decision boundaries around a point.

Original: The performance beneﬁts of this approach are substantial, especially in the case of (cid:96) 2 robustness, where constraint solving is particularly expensive while Euclidean projections can be efﬁciently computed using the dot product and accelerated on
Revised: The performance beneﬁts of this approach are substantial, especially in the case of (cid:96) 2 robustness, where constraint solving is particularly expensive while Euclidean projections can be efﬁciently computed using the dot product and accelerated on

Original: GPU hardware.
Revised: GPU hardware.

Original: However, our approach is also applicable to other norms, including (cid:96) ∞ (Section 3.3).
Revised: However, our approach is also applicable to other norms, including (cid:96) ∞ (Section 3.3).

Original: Our algorithm is embarassingly parallel, and straightforward to implement with facilities for batching that are available in many popular ML libraries.
Revised: Our algorithm is embarassingly parallel, and straightforward to implement with facilities for batching that are available in many popular ML libraries.

Original: Additionally, we show how the algorithm can be easily modiﬁed to ﬁnd certiﬁed lower bounds for (cid:15) , rather than verifying a given ﬁxed value (Section 2.3).
Revised: Additionally, we show how the algorithm can be easily modiﬁed to ﬁnd certiﬁed lower bounds for (cid:15) , rather than verifying a given ﬁxed value (Section 2.3).

Original: Because our algorithm relies exclusively on projections, it may encounter scenarios in which there is evidence to suggest non-robust behavior, but the network’s exact boundaries cannot be conclusively determinedwithoutaccountingforglobalconstraints(Section2, Figure1b).
Revised: Because our algorithm relies exclusively on projections, it may encounter scenarios in which there is evidence to suggest non-robust behavior, but the network’s exact boundaries cannot be conclusively determinedwithoutaccountingforglobalconstraints(Section2, Figure1b).

Original: Insuchcases, thealgorithm will return unknown (though it would be possible to fall back on constraint solving).
Revised: Insuchcases, thealgorithm will return unknown (though it would be possible to fall back on constraint solving).

Original: However, we prove that if the algorithm terminates with a robust decision, then the model satisﬁes Equation 1, and likewise if it returns not _ robust , then an adversarial example exists (Theorem 1).
Revised: However, we prove that if the algorithm terminates with a robust decision, then the model satisﬁes Equation 1, and likewise if it returns not _ robust , then an adversarial example exists (Theorem 1).

Original: Note that unlike prior work on approximate veriﬁcation, our approach can often separate not _ robust cases from unknown , providing a concreteadversarialexampleintheformer.
Revised: Note that unlike prior work on approximate veriﬁcation, our approach can often separate not _ robust cases from unknown , providing a concreteadversarialexampleintheformer.

Original: Inthissense, thealgorithmcanbecharacterizedas sound but incomplete , though our experiments show that in practice the algorithm typically comes to a decision.
Revised: Inthissense, thealgorithmcanbecharacterizedas sound but incomplete , though our experiments show that in practice the algorithm typically comes to a decision.

Original: We show that our implementation outperforms existing exact techniques (Jordan et al., 2019; Tjeng & Tedrake, 2017) by multiple orders of magnitude (Section 3.1, Table 1a and Section 3.3), while rarely being inconclusive on instances for which other techniques do not time out.
Revised: We show that our implementation outperforms existing exact techniques (Jordan et al., 2019; Tjeng & Tedrake, 2017) by multiple orders of magnitude (Section 3.1, Table 2a and Section 3.3), while rarely being inconclusive on instances for which other techniques do not time out.

Original: Moreover, we ﬁnd our approach enables scaling to far deeper models than prior work — a key step towards veriﬁcation of networks that are used in practice.
Revised: Moreover, we ﬁnd our approach enables scaling to far deeper models than prior work — a key step towards veriﬁcation of networks that are used in practice.

Original: Additionally, on models that have been regularized for efﬁcient veriﬁcation (Croce et al., 2019; Xiao et al., 2019), our technique performs even faster, and scales to much larger models — including convolutional networks — than could be veriﬁed using similar techniques (Section 3.1, Table 1a).
Revised: Additionally, on models that have been regularized for efﬁcient veriﬁcation (Croce et al., 2019; Xiao et al., 2019), our technique performs even faster, and scales to much larger models — including convolutional networks — than could be veriﬁed using similar techniques (Section 3.1, Table 2a).

Original: Finally, we compare our work to approximate veriﬁcation methods (Section 3.2).
Revised: Finally, we compare our work to approximate veriﬁcation methods (Section 3.2).

Original: We ﬁnd that while our implementation is not as fast as previous work on efﬁcient lower-bound computation for large models (Weng et al., 2018), our certiﬁed lower bounds are consistently tighter, and in some cases minimal (Section 3.2, Table 1b).
Revised: We ﬁnd that while our implementation is not as fast as previous work on efﬁcient lower-bound computation for large models (Weng et al., 2018), our certiﬁed lower bounds are consistently tighter, and in some cases minimal (Section 3.2, Table 2b).

Original:  A LGORITHM
Revised:  A LGORITHM

Original: In this section we give a high-level overview of our proposed algorithm. We begin with a naive version, which captures the basic exhaustive search (Section 2.1), and then demonstrate how to leverage the compositional structure of the network to optimize the analysis (Section 2.2).
Revised: In this section we give a high-level overview of our proposed algorithm, and present some implementation heuristics that improve its performance (Section 2.2).

Original: We also propose a variant (Section 2.3) to compute certiﬁed lower bounds of the robustness radius.
Revised: We also propose a variant (Section 2.3) to compute certiﬁed lower bounds of the robustness radius.

Original: Correctness proofs for all of the algorithms discussed in this section are provided in Appendix A.We also discuss additional implementation heuristics in Appendix C. Because our algorithm applies to arbitrary (cid:96) p norms, we use the un-subscripted notation (cid:107)·(cid:107) to refer to a general (cid:96) p norm for the remainder of this section.
Revised: Correctness proofs for all of the algorithms discussed in this section are provided in Appendix A. Because our algorithm applies to arbitrary (cid:96) p norms, we use the un-subscripted notation (cid:107)·(cid:107) to refer to a general (cid:96) p norm for the remainder of this section.

Original: 2. T HE B ASIC F AST G EOMETRIC P ROJECTIONS A LGORITHM
Revised: 2. T HE B ASIC F AST G EOMETRIC P ROJECTIONS A LGORITHM

Original: We propose the Fast Geometric Projections (FGP) algorithm, which takes a model, F , an input, x , and a bound, (cid:15) , and either proves that Equation 1 is satisﬁed (i.e., (cid:15) -local robustness), ﬁnds an adversarial input at distance less than (cid:15) from x , or returns unknown .
Revised: We propose the Fast Geometric Projections (FGP) algorithm, which takes a model, F , an input, x , and a bound, (cid:15) , and either proves that Equation 1 is satisﬁed (i.e., (cid:15) -local robustness), ﬁnds an adversarial input at distance less than (cid:15) from x , or returns unknown .

Original: Our algorithm assumes
Revised: Our algorithm assumes

Original: F ( x )= argmax { f ( x ) } , where f : R m → R n is the function computed by a neural network composed of linear transformations with ReLU activations (i.e., f is a feed-forward ReLU network).
Revised: F ( x )= argmax { f ( x ) } , where f : R m → R n is the function computed by a neural network composed of linear transformations with ReLU activations (i.e., f is a feed-forward ReLU network).

Original: The algorithm relies on an analysis of all possible activation regions around x .
Revised: The algorithm relies on an analysis of all possible activation regions around x .

Original: An activation region is a maximal set of inputs having the same activation pattern of ReLU nonlinearities.
Revised: An activation region is a maximal set of inputs having the same activation pattern of ReLU nonlinearities.

Original: Formally, let f u ( x ) denote the pre-activation value of neuron u in network F when evaluating x .
Revised: Formally, let f u ( x ) denote the pre-activation value of neuron u in network F when evaluating x .

Original: We say that neuron u is activated if f u ( x ) ≥ 0 .
Revised: We say that neuron u is activated if f u ( x ) ≥ 0 .

Original: An activation pattern , A , is a Boolean function over neurons that characterizes whether each neuron is activated.
Revised: An activation pattern , A , is a Boolean function over neurons that characterizes whether each neuron is activated.

Original: Then the activation region , ¯ A , associated with pattern A is the set of inputs that realize A : ¯ A :=
Revised: Then the activation region , ¯ A , associated with pattern A is the set of inputs that realize A : ¯ A :=

Original: { x |∀ u. ( f u ( x ) ≥ 0)
Revised: { x |∀ u. ( f u ( x ) ≥ 0)

Original: ⇐⇒
Revised: ⇐⇒

Original: A ( u ) } .
Revised: A ( u ) } .

Original:  x Āx Ā 0 ĀĀ
Revised: Because we assume that F is a piecewise-linear composition of linear transformations and ReLU activations, we can associate the activation status A ( u ) of any neuron with a closed half-space of x Āx Ā 0 ĀĀ

Original: C
Revised: C

Original: Cp x Ā
Revised: Cp x Ā

Original: C B (a) x p Ā
Revised: C B (a) x p Ā

Original: C B a x Ā p
Revised: C B a x Ā p

Original: C B (b)
Revised: C B (b)

Original: Because we assume that F is a piecewise-linear composition of linear transformations and ReLU activations, we can associate the activation status A ( u ) of any neuron with a closed half-space of R m (Jordan et al., 2019).
Revised: R m (Jordan et al., 2019).

Original: The activation constraint , C u , for neuron u and pattern A is the linear inequality w Tu x + b u ≤ 0 , where w u and b u satisfy ∀ x ∈ R m .w
Revised: The activation constraint , C u , for neuron u and pattern A is the linear inequality w Tu x + b u ≤ 0 , where w u and b u satisfy ∀ x ∈ R m .w

Original: Tu x + b ≤ 0 ⇐⇒
Revised: Tu x + b ≤ 0 ⇐⇒

Original: A ( u ) .
Revised: A ( u ) .

Original: The coefﬁcients, w u are equal to the gradient of f u with respect to its inputs, evaluated at a point in ¯ A .
Revised: The coefﬁcients, w u are equal to the gradient of f u with respect to its inputs, evaluated at a point in ¯ A .

Original: Crucially, because the gradient is the same at every point in ¯ A , the constraints can be computed from the activation pattern alone via backpropagation.
Revised: Crucially, because the gradient is the same at every point in ¯ A , the constraints can be computed from the activation pattern alone via backpropagation.

Original: More details on this computation are given in Appendix B.
Revised: More details on this computation are given in Appendix B.

Original: The intersection of these constraints yields the activation region ¯ A , and the facets of ¯ A correspond to the non-redundant constraints.
Revised: The intersection of these constraints yields the activation region ¯ A , and the facets of ¯ A correspond to the non-redundant constraints.

Original: The convexity of activation regions follows from this observation, as does the fact that the decision boundaries are also linear constraints of the form f( x ) i ≥ f ( x ) j for classes
Revised: The convexity of activation regions follows from this observation, as does the fact that the decision boundaries are also linear constraints of the form f

Original: i and j . The FGP algorithm performs a search of all activation regions that might be at distance less than (cid:15) from x .
Revised: The FGP algorithm performs a search of all activation regions that might be at distance less than (cid:15) from x .

Original: We begin by analyzing the region, ¯ A 0 , associated with the activation pattern of the input, x , as follows.
Revised: We begin by analyzing the region, ¯ A 0 , associated with the activation pattern of the input, x , as follows.

Original: First, we check to see if ¯ A 0 contains a decision boundary that is at distance less than (cid:15) from x ; if so, we take the projection, p , from x onto C .
Revised: First, we check to see if ¯ A 0 contains a decision boundary, C , that is at distance less than (cid:15) from x ; if so, we take the projection, p , from x onto C .

Original: Intuitively, the projection is an input satisfying C that is at minimal distance from x , i.e., ∀ x (cid:48) .C
Revised: Intuitively, the projection is an input satisfying C that is at minimal distance from x , i.e., ∀ x (cid:48) .C

Original: ( x (cid:48) )
Revised: ( x (cid:48) )

Original: =
Revised: =

Original: ⇒(cid:107)
Revised: ⇒(cid:107)

Original: x − p (cid:107)≤(cid:107) x − x (cid:48) (cid:107) .
Revised: x − p (cid:107)≤(cid:107) x − x (cid:48) (cid:107) .

Original: Similarly we deﬁne the distance, d , from x to a constraint or decision boundary, C , as d ( x,C ):=
Revised: Similarly we deﬁne the distance, d , from x to a constraint or decision boundary, C , as d ( x,C ):=

Original: min x (cid:48) :
Revised: min x (cid:48) :

Original: C ( x (cid:48) )
Revised: C ( x (cid:48) )

Original: (cid:107) x − x (cid:48) (cid:107) , i.e., d is the distance from x to its projection onto C .
Revised: (cid:107) x − x (cid:48) (cid:107) , i.e., d is the distance from x to its projection onto C .

Original: If p does not have the same class as x according to F (or lies directly on the decision boundary between classes), then p is an adversarial example.
Revised: If p does not have the same class as x according to F (or lies directly on the decision boundary between classes), then p is an adversarial example.

Original: If p has the same class as x , it means that the projection to the decision boundary is outside of the activation region we are currently analyzing; however, this is not sufﬁcient to conclude that no point on the decision boundary is inside both the current activation region and the (cid:15) -ball
Revised: If p has the same class as x , it means that the projection to the decision boundary is outside of the activation region we are currently analyzing; however, this is not sufﬁcient to conclude that no point on the decision boundary is inside both the current activation region and the (cid:15) -ball

Original: (see Figure 1b).
Revised: (see Figure 1b).

Original: Therefore, we return unknown .
Revised: Therefore, we return unknown .

Original: Otherwise, we collect all activation constraints in ¯ A 0 ; there is one such activation constraint per neuron u in the network; thus each constraint corresponds to the neighboring region, ¯ A u 0 , which has the same activation pattern as ¯ A 0 , except with neuron u ﬂipped, i.e., A 0 ( u (cid:48) )
Revised: Otherwise, we collect all activation constraints in ¯ A 0 ; there is one such activation constraint per neuron u in the network; thus each constraint corresponds to the neighboring region, ¯ A u 0 , which has the same activation pattern as ¯ A 0 , except with neuron u ﬂipped, i.e., A 0 ( u (cid:48) )

Original: (cid:54) =
Revised: (cid:54) =

Original: A u 0 ( u (cid:48) )
Revised: A u 0 ( u (cid:48) )

Original: ⇐⇒ u
Revised: ⇐⇒ u

Original: = u (cid:48) .
Revised: = u (cid:48) .

Original: For each constraint  at distance less than (cid:15) from x  , we enqueue the corresponding activation region if it has not been searched yet.
Revised: For each constraint C u , we check whether it might be at distance less than (cid:15) from x using a geometric projection. Although C u is only valid in the polytope corresponding to the activation region ¯ A 0 , we compute the distance to the hyperplane corresponding to the inﬁnite prolongation of C u . As such, we only obtain a lower bound on the distance from x to C u . If the computed distance is smaller than (cid:15) , we enqueue the corresponding activation region if it has not been searched yet.

Original: Thus the queue contains unexplored regions that might be at distance less than (cid:15) from x .
Revised: Thus the queue contains unexplored regions that might be at distance less than (cid:15) from x .

Original: The algorithm continues to analyze each enqueued region in the same way until the queue is empty.
Revised: The algorithm continues to analyze each enqueued region in the same way, each explored region enqueuing neighbouring regions that might be at distance less than (cid:15) from x , until the queue is empty.

Original: Exhausting the queue means that we did not ﬁnd any adversarial examples in any activation region that might intersect with the (cid:15) -ball centered at x .
Revised: Exhausting the queue means that we did not ﬁnd any adversarial examples in any activation region that might intersect with the (cid:15) -ball centered at x .

Original: We therefore conclude that x is locally (cid:15) -robust.
Revised: We therefore conclude that x is locally (cid:15) -robust.

Original: An illustration of a simple execution of this algorithm is shown in Figure 1a.
Revised: An illustration of a simple execution of this algorithm is shown in Figure 1a.

Original: Conditions on Adversarial Examples.
Revised: Conditions on Adversarial Examples.

Original: It is common in practice to include domain-speciﬁc conditions specifying which inputs are considered valid adversarial examples.
Revised: It is common in practice to include domain-speciﬁc conditions specifying which inputs are considered valid adversarial examples.

Original: For example, in the image domain, adversarial examples are typically subject to box constraints requiring each pixel to  be in the range [0 , 1] .
Revised: For example, in the image domain, adversarial examples are typically subject to box constraints requiring each pixel to [Table] be in the range [0 , 1] .

Original: These constraints can easily be incorporated into the FGP algorithm by checking [-1 off, 2 off 1 on, 2 off 1 off, 2 on 1 on, 2 on 3 off, 4 off 3 on, 4 off 3 off, 4 on 3 on, 4 on 00000011100010110100011111001111(a) Ā 
 ĀĀĀ-]
Revised: These constraints can easily be incorporated into the FGP algorithm by checking [-1 off, 2 off 1 on, 2 off 1 off, 2 on 1 on, 2 on 3 off, 4 off 3 on, 4 off 3 off, 4 on 3 on, 4 on 00000011100010110100011111001111(a) Ā 
 ĀĀĀ-]them when we check if F ( x ) (cid:54)

Original: C C LCA C(b) them when we check if F ( x ) (cid:54) =
Revised: =

Original: F ( p ) , when a decision boundary is found.
Revised: F ( p ) , when a decision boundary is found.

Original: If p is not a valid adversarial example, the algorithm would return unknown 
Revised: If p is not a valid adversarial example, the algorithm would return unknown .

Original: Correctness.
Revised: Correctness.

Original: We show that when the FGP algorithm returns not _ robust , there exists an adversarial example, and when it returns robust , the model is locally robust at x , with radius (cid:15) .
Revised: We show that when the FGP algorithm returns not _ robust , there exists an adversarial example, and when it returns robust , the model is locally robust at x , with radius (cid:15) .

Original: However, the algorithm may also return unknown , in which case we do not claim anything about the robustness of the model.
Revised: However, the algorithm may also return unknown , in which case we do not claim anything about the robustness of the model.

Original: A complete proof of Theorem 1 is provided in Appendix A.1.
Revised: A complete proof of Theorem 1 is provided in Appendix A.1.

Original: Theorem 1.
Revised: Theorem 1.

Original: (1) When the FGP algorithm returns not _ robust , there exists an adversarial example, p , such that (cid:107) x − p (cid:107)≤ (cid:15) and F ( x ) (cid:54) =
Revised: (1) When the FGP algorithm returns not _ robust , there exists an adversarial example, p , such that (cid:107) x − p (cid:107)≤ (cid:15) and F ( x ) (cid:54) =

Original: F ( p ) .
Revised: F ( p ) .

Original: (2) When it returns robust , Equation 1 holds.
Revised: (2) When it returns robust , Equation 1 holds.

Original: L EVERAGING THE C OMPOSITIONAL S TRUCTURE OF F EED - FORWARD N ETWORKS
Revised: 2. H EURISTICS FOR FGP

Original: The algorithm presented in Section 2.1 searches a loose approximation of the set of regions contained in the (cid:15) -ball about x . In this section we demonstrate how the compositional structure of the network can be leveraged to narrow the search space.
Revised: To improve the performance and precision of our algorithm, we use several additional heuristics.

Original: As mentioned previously, our focus is on feed-forward ReLU networks. Such networks contain an ordered hierarchy of layers. Therefore, for a neuron u , the activation constraint corresponding to the condition f u ( x ) > 0 only depends on the activations of neurons in previous layers. In particular, if we ﬁx the activations of all neurons at previous layers, then this constraint becomes linear.
Revised: Batching.

Original: Our crucial observation is that, if two activations patterns A andA (cid:48) differ only in the activation of one neuron, u , in layer N , the activation constraints for all neurons up to layer N remain identical in A and A (cid:48) , as they do not depend onneurons deeper in the network such as u . We leverage this hierarchy to enhance FGP with a tree-based exploration of the network to prove that an input x is (cid:15) -robust. Each node in our exploration tree corresponds to the activation pattern of neurons at a layer.
Revised: The gradient backpropagations for computing constraints and the dot product projections for calculating distances lend themselves well to batching on the GPU. In our implementation we dequeue multiple elements from the queue at once and calculate the constraints and constraint distances for the corresponding regions in parallel, leading to a speedup of up to 2x on large examples for a well-chosen batch size ( ∼ 10-100).

Original: Nodes are ordered similarly to the layers in the network: if layer N has k neurons, the vertices at depth N − 1 have degree 2 k , with one child for each possible activation pattern of these k neurons. Leaf nodes give a complete characterization of an activation pattern for the entire network. Leaves that share an ancestor at depth N in the tree correspond to activation patterns that share an activation preﬁx , i.e., the activations of all neurons in the layers up to layer N are the same in both activation patterns. An illustration of such a tree is shown in Figure 2a.
Revised: Caching Intermediate Computations.

Original: The version of the FGP algorithm given in Section 2.1 systematically explores all regions that might be at distance less than (cid:15) from x ; a region ¯ A i is explored if it is reachable from a previously explored region (starting with the one containing x ) by ﬂipping the activation of exactly one neuron.
Revised: We observe that activation constraints for the ﬁrst layer are invariant across all regions, as there is no earlier neuron to depend on. Thus, we can precompute all such constraints once for all inputs to be analyzed.

Original: Exploration of ¯
Revised: Exploring the Full Queue.

Original: A i consists of collecting all activation constraints and decision boundaries at distance less than (cid:15) from x w.r.t the activation pattern A i , and enqueuing the corresponding neighboring regions to analyze them.
Revised: If the analysis of a decision boundary is inconclusive, the FGP algorithm, as presented, stops and returns unknown .

Original: In our optimized variant, when enqueuing a region, ¯ A i +1 , we additionally keep track of its predecessor in the exploration, ¯ A i , i.e., the region from which ¯
Revised: We also implement a variant where we instead record the presence of an ambiguous decision boundary but continue exploring other regions in the queue.

Original: A i +1 was discovered. Crucially, when exploring ¯ A i +1 , we only enqueue neighboring regions belonging to the subtree of the lowest common ancestor (LCA) of ¯
Revised: If we empty the queue after unsuccessfully analyzing a decision boundary, we nevertheless return unknown .

Original: A i and ¯ A
Revised: However, oftentimes a true adversarial example is found during the continued search, allowing us to return not _ robust conclusively.

Original: i +1 in the tree. Supposing neuron u was ﬂipped to obtain A i +1 from A i , this corresponds to considering only activation constraints in ¯ A i +1 that are associated with the layer u belongs to or deeper. To understand how this approach helps prune unnecessary regions that the basic FGP algorithm would otherwise have searched, an example is provided in Figure 2b.
Revised: Table 1 displays the results of experiments evaluating the impact of exploring the entire search queue instead of stopping at the ﬁrst decision boundary (as described in Section 2.1).

Original: A proof of  the correctness ofthis optimized variant is provided in Appendix A.2. In practice, for efﬁciency, we do not construct the tree in memory.
Revised: The experimental results show that this heuristic decreases the number of unknown results by approximately 50% while only having a minor impact on the speed of the execution.

Original: Instead, it is sufﬁcient to use a regular queue of regions toexplore, enriched with information about which layer the last ﬂipped neuron belongs to. When exploring a region, we ﬁlter out all collected constraints that belong to earlier layers. This preserves the inherent parallelism from the basic version of FGP, leading to a more efﬁcient implementation.
Revised: Moreover, we reduce the number of unknown results to the extent that we recover the results obtained by GeoCert in every instance for which GeoCert terminates, except 3 non-robust points on mnist40x3, while nevertheless performing our analysis two to three orders of magnitude more quickly.

Original: 2. C ERTIFIED L OWER B OUNDS
Revised: 2. C ERTIFIED L OWER B OUNDS

Original: We now consider the related problem of ﬁnding a lower bound on the local robustness of the network.
Revised: We now consider the related problem of ﬁnding a lower bound on the local robustness of the network.

Original: A variant of the FGP algorithm can provide certiﬁed lower bounds by using a priority queue; constraints are enqueued with priority corresponding to their distance from x , such that the closest constraint is always at the front of the queue.
Revised: A variant of the FGP algorithm can provide certiﬁed lower bounds by using a priority queue; constraints are enqueued with priority corresponding to their distance from x , such that the closest constraint is always at the front of the queue.

Original: We keep track of the current certiﬁed lower bound in a variable, β .
Revised: We keep track of the current certiﬁed lower bound in a variable, β .

Original: At each iteration, we set β to the maximum of the old β and the distance from x to the constraint at the front of the queue.
Revised: At each iteration, we set β to the maximum of the old β and the distance from x to the constraint at the front of the queue.

Original: The algorithm terminates either when all constraints at distance less than the initially speciﬁed (cid:15) were handled, or when a decision boundary for which the initial algorithm returns unknown or not _ robust is found.
Revised: The algorithm terminates either when all constraints at distance less than the initially speciﬁed (cid:15) were handled, or when a decision boundary for which the initial algorithm returns unknown or not _ robust is found.

Original: In the ﬁrst case, we return (cid:15) ; in the second, we return the value stored in β at this iteration.
Revised: In the ﬁrst case, we return (cid:15) ; in the second, we return the value stored in β at this iteration.

Original: The proof of this variant is similar to the proof of the FGP algorithm.
Revised: The proof of this variant is similar to the proof of the FGP algorithm.

Original: It relies on the following loop invariant, which we prove in Appendix A.3.
Revised: It relies on the following loop invariant, which we prove in Appendix A.2.

Original: Loop Invariant 1.
Revised: Loop Invariant 1.

Original: (1) All activation regions at distance less than β from x were previously visited, (2) β is always smaller than (cid:15) , and (3) there is no adversarial point at distance less than β .
Revised: (1) All activation regions at distance less than β from x were previously visited, (2) β is always smaller than (cid:15) , and (3) there is no adversarial point at distance less than β .

Original:  E VALUATION
Revised:  E VALUATION

Original: In this section, we evaluate the performance of our implementation of the FGP algorithm and its variant for computing lower bounds.
Revised: In this section, we evaluate the performance of our implementation of the FGP algorithm and its variant for computing lower bounds.

Original: Section 3.1 discusses the performance of FGP compared to existing tools that perform exact (cid:96) 2 robustness veriﬁcation.
Revised: Section 3.1 discusses the performance of FGP compared to existing tools that perform exact (cid:96) 2 robustness veriﬁcation.

Original: Section 3.2 compares the lower bounds certiﬁed by our
Revised: Section 3.2 compares the lower bounds certiﬁed by our

Original: FGP variant from Section 2.3 to those of other approximate certiﬁcation methods.
Revised: FGP variant from Section 2.3 to those of other approximate certiﬁcation methods.

Original: In short, we ﬁnd that our approach outperforms existing exact veriﬁcation tools by several orders of magnitude (2-4), and produces more precise lower bounds (3-25 times larger) than the relevant work on approximate certiﬁcation.
Revised: In short, we ﬁnd that our approach outperforms existing exact veriﬁcation tools by several orders of magnitude (2-4), and produces more precise lower bounds (3-25 times larger) than the relevant work on approximate certiﬁcation.

Original: Additionally, in Section 3.1, we explore the scalability of our algorithm — particularly with respect to the depth of the network being veriﬁed, ﬁnding that pruning based on the network’s compositional structure (Section 2.2) is particularly useful for scaling to deeper networks.
Revised: Additionally, in Section 3.1, we explore the scalability of our algorithm.

Original: Furthermore, we ﬁnd that when the networks are strongly regularized for veriﬁability, our approach even scales to CNNs.
Revised: We ﬁnd that when the networks are strongly regularized for veriﬁability, our approach even scales to CNNs.

Original: Finally, weremarkontheﬂexibilityofFGPwithrespecttothenorminSection3.3, andobservethatourapproach is also faster than existing complete veriﬁcation tools when performing (cid:96) ∞ robustness veriﬁcation.
Revised: Finally, we remark on the ﬂexibility of FGP with respect to the norm in Section 3.3, and observe that our approach is also faster than existing complete veriﬁcation tools when performing (cid:96) ∞ robustness veriﬁcation.

Original: We performed experiments on three standard datasets: MNIST, Fashion-MNIST, and CIFAR10.
Revised: We performed experiments on three standard datasets: MNIST, Fashion-MNIST, and CIFAR10.

Original: We evaluated both on models trained for robustness using adversarial training (Madry et al., 2018), and on models trained for veriﬁability and robustness using maximum margin regularization (MMR)
Revised: We evaluated both on models trained for robustness using adversarial training (Madry et al., 2018), and on models trained for veriﬁability and robustness using maximum margin regularization (MMR)

Original: (Croce et al., 2019) or ReLU Stability (RS) (Xiao et al., 2019).
Revised: (Croce et al., 2019) or ReLU Stability (RS) (Xiao et al., 2019).

Original: We refer to each model by the dataset it was trained on, followed by the architecture of the hidden layers.
Revised: We refer to each model by the dataset it was trained on, followed by the architecture of the hidden layers.

Original: For example, “mnist20x3” refers to a model trained on MNIST with 3 hidden layers of 20 neurons each.
Revised: For example, “mnist20x3” refers to a model trained on MNIST with 3 hidden layers of 20 neurons each.

Original: The “cnn” architecture refers to a common CNN architecture used to benchmark CNN veriﬁcation in the literature; details are given in Appendix D. Models marked with “*” were trained with MMR; models marked with “ † ” were trained with RS; all other models were trained using PGD adversarial training (Madry et al., 2018).
Revised: The “cnn” architecture refers to a common CNN architecture used to benchmark CNN veriﬁcation in the literature; details are given in Appendix C. Models marked with “*” were trained with MMR; models marked with “†” were trained with RS; all other models were trained using PGD adversarial training (Madry et al., 2018).

Original: The hyperparameters used for training are given in Appendix D. [Table]
Revised: The hyperparameters used for training are given in Appendix C.

Original: In each experiment, measurements are obtained by evaluating on 100 arbitrary instances from the test set.
Revised: In each experiment, measurements are obtained by evaluating on 100 arbitrary instances from the test set.

Original: All experiments were run on a 4.2GHz Intel Core i7-7700K with 32 GB of RAM, and a Tesla
Revised: All experiments were run on a 4.2GHz Intel Core i7-7700K with 32 GB of RAM, and a Tesla

Original: K80 GPU with 12 GB of RAM.
Revised: K80 GPU with 12 GB of RAM.

Original: 3. L OCAL R OBUSTNESS C ERTIFICATION
Revised: 3. L OCAL R OBUSTNESS C ERTIFICATION

Original: We ﬁrst compare the efﬁciency of our implementation to that of other tools certifying local robustness.
Revised: We ﬁrst compare the efﬁciency of our implementation to that of other tools certifying local robustness.

Original: GeoCert (Jordan et al., 2019) and MIP (Tjeng & Tedrake, 2017) are the tools most comparable to ours as they are able to exactly check for robustness with respect to the (cid:96) 2 norm.
Revised: GeoCert (Jordan et al., 2019) and MIP (Tjeng & Tedrake, 2017) are the tools most comparable to ours as they are able to exactly check for robustness with respect to the (cid:96) 2 norm.

Revised: We used commit hash 8730aba for GeoCert, and v0.2.1 for MIP.

Original: Speciﬁcally, we compare the median run time over the analyses of each of the selected 100 instances, the number of examples on which each tool terminates, the result of the analyses that terminate, and the corresponding veriﬁed robust accuracy (VRA), i.e., the fraction of points that were both correctly classiﬁed by the model and veriﬁed as robust.
Revised: Speciﬁcally, we compare the median run time over the analyses of each of the selected 100 instances, the number of examples on which each tool terminates, the result of the analyses that terminate, and the corresponding veriﬁed robust accuracy (VRA), i.e., the fraction of points that were both correctly classiﬁed by the model and veriﬁed as robust.

Original: In addition, we report an upper bound on the VRA for each model, obtained by running PGD attacks (hyperparameters included in Appendix D) on the correctly-classiﬁed points on which every method either timed out or reported unknown .
Revised: In addition, we report an upper bound on the VRA for each model, obtained by running PGD attacks (hyperparameters included in Appendix C) on the correctly-classiﬁed points on which every method either timed out or reported unknown .

Original: Results are given for an (cid:96) 2 norm bound of (cid:15) =0 .
Revised: Results are given for an (cid:96) 2 norm bound of (cid:15) =0 .

Original: 25 , with a computation budget of two minutes per instance.
Revised: 25 , with a computation budget of two minutes per instance.

Original: The results for these experiments are presented in Table 1a.
Revised: The results for these experiments are presented in Table 2a.

Original: We observe that FGP always outperforms GeoCert by two to three orders of magnitude, without sacriﬁcingprecision; i.e., werarelyreturn unknown whenGeoCertterminates.
Revised: We observe that FGP always outperforms GeoCert by two to three orders of magnitude, without sacriﬁcing precision; i.e., we rarely return unknown when GeoCert terminates.

Original: MIPfrequentlytimesout with a time budget of 120 seconds, indicating that we are faster by at least four orders of magnitude.
Revised: MIP frequently times out with a time budget of 120 seconds, indicating that we are faster by at least four orders of magnitude.

Original: This is consistent with the 100 to 1000 seconds per solve on a MNIST model with three hidden layers ofneurons each reported by Tjeng & Tedrake (2017); their technique performs best on the (cid:96) 1 or (cid:96) ∞ norms.
Revised: This is consistent with the 100 to 1000 seconds per solve on a MNIST model with three hidden layers of 20 neurons each reported by Tjeng & Tedrake (2017); their technique performs best on the (cid:96) 1 or (cid:96) ∞ norms.

Original: In addition, we ﬁnd that FGP consistently veriﬁes the highest fraction of points, and yields the best VRA.
Revised: In addition, we ﬁnd that FGP consistently veriﬁes the highest fraction of points, and yields the best VRA.

Original: Moreover, on mnist20x3, FGP comes within 1% of the VRA upper bound; on mnist20x6 it comes within 4% .
Revised: Moreover, on mnist20x3, FGP comes within 2 percentage points of the VRA upper bound.

Original: This suggests that FGP comes close to verifying every robust instance on these models.
Revised: This suggests that FGP comes close to verifying every robust instance on this model. [Table]

Revised: fmnist1000x 0.

Revised: (b)

Original: Models Trained for Veriﬁability.
Revised: Models Trained for Veriﬁability.

Original: Adversarial training (Madry et al., 2018), used to train the models in Sections 3.1 and 3.2, attempts to ensure that points on the data manifold will be far from the model’s decision boundary.
Revised: Adversarial training (Madry et al., 2018), used to train the models in Sections 3.1 and 3.2, attempts to ensure that points on the data manifold will be far from the model’s decision boundary.

Original: Even if it achieves this objective, it may nonetheless be difﬁcult to verify because the performance of FGP depends not only on the decision boundaries, but also on the internal activation constraints.
Revised: Even if it achieves this objective, it may nonetheless be difﬁcult to verify because the performance of FGP depends not only on the decision boundaries, but also on the internal activation constraints.

Original: Thus, we expect certiﬁcation will be efﬁcient only when the points to be certiﬁed are also far from internal boundaries, leading to fewer regions to explore.
Revised: Thus, we expect certiﬁcation will be efﬁcient only when the points to be certiﬁed are also far from internal boundaries, leading to fewer regions to explore.

Original: Recent work has sought to develop training procedures that not only encourage robustness, but that also optimize for efﬁcient veriﬁcation (Croce et al., 2019; Wong & Kolter, 2018; Xiao et al., 2019).
Revised: Recent work has sought to develop training procedures that not only encourage robustness, but that also optimize for efﬁcient veriﬁcation (Croce et al., 2019; Wong & Kolter, 2018; Xiao et al., 2019).

Original: Maximum margin regularization (MMR) (Croce et al., 2019) and ReLU stability (RS) (Xiao et al., 2019) are particularly suited to our work, as they follow from the same intuition highlighted above.
Revised: Maximum margin regularization (MMR) (Croce et al., 2019) and ReLU stability (RS) (Xiao et al., 2019) are particularly suited to our work, as they follow from the same intuition highlighted above.

Original: [Figure] Using MMR, FGP is able to scale to much larger models with hundreds or thousands of neurons, and tens of layers, as shown in the bottom half of Table 1a.
Revised: Using MMR, FGP is able to scale to much larger models with hundreds or thousands of neurons, and tens of layers, as shown in the bottom half of Table 2a.

Original: Here again, we see that FGP outperforms the other approaches in terms of both time and successfully-veriﬁed points.
Revised: Here again, we see that FGP outperforms the other approaches in terms of both time and successfully-veriﬁed points.

Original: By comparison, while GeoCert also experienced improved performance on the MMR-trained models, our method remains over three orders of magnitude faster.
Revised: By comparison, while GeoCert also experienced improved performance on the MMR-trained models, our method remains over three orders of magnitude faster.

Original: MIP continued to time out in nearly every case.
Revised: MIP continued to time out in nearly every case.

Original: We found that we were able provide even stronger regularization with RS, allowing us to scale to CNNs (Table 1a), which have far more internal neurons than even large dense networks.
Revised: We found that we were able provide even stronger regularization with RS, allowing us to scale to CNNs (Table 2a), which have far more internal neurons than even large dense networks.

Original: We found that these highly regularized CNNs veriﬁed more quickly than some of the less-regularized dense networks, though, as with other methods that produce veriﬁable CNNs, this came at the cost of a penalty on the model’s accuracy.Depth Scalability.
Revised: We found that these highly regularized CNNs veriﬁed more quickly than some of the less-regularized dense networks, though, as with other methods that produce veriﬁable CNNs, this came at the cost of a penalty on the model’s accuracy.

Original: FGP searches an overapproximation of the relevant search space for an adversarial example, potentially searching many more regions than are necessary to certify robustness.

Original: Figureplots the number of regions searched by FGP, as compared to those searched by GeoCert, which searches the minimal required set of regions.

Original: In a single-layer model, FGP searches 10x as many regions as GeoCert, while in the deep model with 10 layers it searches only 3x as many regions, suggesting that FGP becomes particularly efﬁcient on deeper networks .

Original: Note that the cost per region is more than 2,000x less in FGP than in GeoCert.

Original: 3. C ERTIFIED L OWER B OUNDS
Revised: 3. C ERTIFIED L OWER B OUNDS

Original: We now evaluate the variant of our algorithm computing certiﬁed lower bounds on the robustness radius (Section 2.3).
Revised: We now evaluate the variant of our algorithm computing certiﬁed lower bounds on the robustness radius (Section 2.3).

Original: To this end, we compare the performance of our approach to FastLin (Weng et al., 2018), which is designed to provide quick, but potentially loose lower bounds on the local robustness of large ReLU networks.
Revised: To this end, we compare the performance of our approach to FastLin (Weng et al., 2018), which is designed to provide quick, but potentially loose lower bounds on the local robustness of large ReLU networks.

Original: We compare the certiﬁed lower bound reported by our implementation of FGP after 60 seconds of computation (on models large enough such that FGP rarely terminates in 60 seconds) to the lower bound reported by FastLin;  the  results are presented in Table 1b.
Revised: We compare the certiﬁed lower bound reported by our implementation of FGP after 60 seconds of computation (on models large enough such that FGP rarely terminates in 60 seconds) to the lower bound reported by FastLin; FastLin always reported results in less than two seconds on the models analyzed. The results are presented in Table 2b.

Original: The mean lower bound is reported for both methods, and we observe that on the models tested, FGP is able to ﬁnd a better lower bound on average, though it requires considerably more computation time.
Revised: The mean lower bound is reported for both methods, and we observe that on the models tested, FGP is able to ﬁnd a better lower bound on average, though it requires considerably more computation time.

Original: Because the optimal bound may vary between instances, we also report the median ratio of the lower bounds obtained by the two methods on each individual instance.
Revised: Because the optimal bound may vary between instances, we also report the median ratio of the lower bounds obtained by the two methods on each individual instance.

Original: Here we see that FastLin may indeed  be quite loose, as on a typical instance it achieves as low as 4% and only as high as 69% of the bound obtained by FGP.
Revised: Here we see that FastLin may indeed [Table] be quite loose, as on a typical instance it achieves as low as 4% and only as high as 69% of the bound obtained by FGP.

Original: Finally, we note that when FGP terminates by ﬁnding a decision boundary, if the projection onto that boundary is a true adversarial example, then the lower bound is tight .
Revised: Finally, we note that when FGP terminates by ﬁnding a decision boundary, if the projection onto that boundary is a true adversarial example, then the lower bound is tight .

Original: In our experiments, there were few such examples — three on fmnist500x4 and one on fmnist1000x3 — however, in these cases, the lower bound obtained by FastLin was very loose, achieving 4-15% of the optimal bound on fmnist500x4, and only 0.8% of the optimal bound on fmnist1000x3.
Revised: In our experiments, there were few such examples — three on fmnist500x4 and one on fmnist1000x3 — however, in these cases, the lower bound obtained by FastLin was very loose, achieving 4-15% of the optimal bound on fmnist500x4, and only 0.8% of the optimal bound on fmnist1000x3.

Original: This suggests that while FastLin has been demonstrated to scale to large networks, one must be careful with its application, as there may be cases in which the bound it provides is too conservative.
Revised: This suggests that while FastLin has been demonstrated to scale to large networks, one must be careful with its application, as there may be cases in which the bound it provides is too conservative.

Original: 3. G ENERALIZATION TO O THER N ORMS
Revised: 3. G ENERALIZATION TO O THER N ORMS

Original: The only aspect of FGP that depends on the norm is the projection and the projected distance computation, making our approach highly modular with respect to the norm.
Revised: The only aspect of FGP that depends on the norm is the projection and the projected distance computation, making our approach highly modular with respect to the norm.

Revised: As such, we added support for (cid:96) ∞ robustness certiﬁcation by providing a projection and a projected distance computation in FGP.

Original: We evaluated an implementation of FGP using the (cid:96) ∞ norm against GeoCert (Jordan et al., 2019), anda complete version of ERAN (Singh et al., 2019a) that falls back on constraint-solving when abstract interpretation yields an inconclusive result.
Revised: In this section, we compare the efﬁciency of this implementation to GeoCert (Jordan et al., 2019) and ERAN (Singh et al., 2019a). ERAN uses abstract interpretation to analyze networks using conservative over-approximations.

Original: Full experimental results are available in 
Revised: Assuch, theanalysiscancertifythataninputisrobust, butcanyieldfalsepositives when ﬂagging an input as not robust. In this evaluation, we use the DeepPoly abstract domain, and the complete mode of ERAN which falls back on constraint solving when an input is classiﬁed as not robust. Our experimental setup is similar to the one presented in Section 3.1. For each network, we arbitrarily pick 100 inputs from the test set and report the median analysis time of each tool.

Original: Appendix E. We observed that (1) the performance gain compared to constraint-solving was less signiﬁcant than in the (cid:96) 2 case, and (2) the number of unknown cases was higher compared with the (cid:96) 2 implementation.
Revised: We give each tool a time budget of 120 seconds per input. We report the number of instances on which each tool terminates within this time frame, as well as the results of the analyses that terminate. Results are given in Tablefor an (cid:96) ∞ norm bound of (cid:15) =0 . 01 , which contains roughly the same volume as the (cid:96) 2 ball used in most of our evaluation.

Original: We suspect that the former is because constraint-solving is more efﬁcient in the (cid:96) ∞ space (as the objective function is linear, and interval-bound propagation is ofgreater use), and that the latter suggests projection-based distances are a better approximation in Euclidean space than in (cid:96) ∞ space.
Revised: We refer to each model by the dataset it was trained on, followed by the architecture of the hidden layers. For example, “mnist20x3” refers to a model trained on MNIST with 3 hidden layers ofneurons each. Models marked with an asterisk were trained with MMR (Croce et al., 2019); other models were trained using PGD adversarial training (Madry et al., 2018).

Original: We nevertheless consistently outperformed
Revised: We observe that we consistently outperform GeoCert by about two orders of magnitude, both on models trained with MMR and using adversarial training. On models trained for robustness (upper rows of Table 3), FGP is 0.5x to 5x faster than ERAN. In particular, FGP seems to scale better to deeper networks, while ERAN performs better on wider networks. Interestingly, ERAN does not beneﬁt as much from MMR training as FGP and GeoCert.

Original: GeoCert by about two orders of magnitude, and were up to 5x faster than ERAN on models trained with PGD, and one to two orders of magnitude faster than  ERAN on models trained using MMR.
Revised: On large models trained with MMR, our tool is one to two orders of magnitude faster than ERAN, and ERAN is only about 2x faster than GeoCert.

Revised: Finally, while FGP is able to determine robustness for almost all inputs deemed robust by GeoCert or ERAN, it struggles to ﬁnd adversarial examples for non-robust points, which results in a higher number of unknown cases compared to the (cid:96) 2 variant.

Revised: This is not highly surprising, as projections are better-suited to the Euclidean space than to (cid:96) ∞ space.

Revised: While the projection in Euclidean space is unique, this is not always the case in (cid:96) ∞ space; we pick one arbitrary projection which thus might not lead to an adversarial example.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work can be grouped with approaches for verifying neural networks that aim to check local robustness exactly (Jordan et al., 2019; Katz et al., 2017; 2019; Tjeng & Tedrake, 2017); the primary difference is that our approach avoids expensive constraint solving at the price of incompleteness.
Revised: Our work can be grouped with approaches for verifying neural networks that aim to check local robustness exactly (Jordan et al., 2019; Katz et al., 2017; 2019; Tjeng & Tedrake, 2017); the primary difference is that our approach avoids expensive constraint solving at the price of incompleteness.

Original: GeoCert (Jordan et al., 2019) is the closest work to ours; it aims to exactly compute local robustness of deep neural networks for convex norms.
Revised: GeoCert (Jordan et al., 2019) is the closest work to ours; it aims to exactly compute local robustness of deep neural networks for convex norms.

Original: Unlike our approach, GeoCert computes the largest (cid:96) p ball centered at an input point within which the network is robust.
Revised: Unlike our approach, GeoCert computes the largest (cid:96) p ball centered at an input point within which the network is robust.

Original: Our experimental comparison with GeoCert shows that our approach scales much better.
Revised: Our experimental comparison with GeoCert shows that our approach scales much better.

Original: This is not surprising as GeoCert relies on projections to polytopes, which are solved by a quadratic program (QP) with linear constraints.
Revised: This is not surprising as GeoCert relies on projections to polytopes, which are solved by a quadratic program (QP) with linear constraints.

Revised: This enables to compute exactly the distance from a point to an activation constraint, instead of an underapproximation as our approach does.

Original: In contrast, our approach uses projections to afﬁne subspaces, which have a simpler, closed-form solution.
Revised: In contrast, our approach uses projections to afﬁne subspaces, which have a simpler, closed-form solution.

Original: MIP (Tjeng & Tedrake, 2017) is an alternative to GeoCert based on mixed-integer programming; it also requires solving QPs for the (cid:96) 2 norm.

Original: We could fall back on similar techniques to provide a slower, but complete variant of our algorithm when our projections cannot reach a conclusion about a decision boundary.

Original: Lim et al.
Revised: Lim et al.

Original: (2020)is concurrent work that builds on GeoCert by using a heuristic similar to ours presented in Section 2.2.
Revised: (2020) builds on GeoCert, but reduces the number of activation regions to consider by exploring the network in a hierarchical manner.

Original: Although Lim et al. reduce the number of QPs required for veriﬁcation, their variant still relies on constraint solving.
Revised: Although Lim et al. reduce the number of QPs required for veriﬁcation, their variant still relies on constraint solving.

Original: While they improve on GeoCert by up to a single order of magnitude, our approach consistently outperforms GeoCert by 2-4 orders of magnitude.
Revised: While they improve on GeoCert by up to a single order of magnitude, our approach consistently outperforms GeoCert by 2-4 orders of magnitude.

Revised: MIP (Tjeng & Tedrake, 2017) is an alternative to GeoCert based on mixed-integer programming; it also requires solving QPs for the (cid:96) 2 norm.

Revised: We could fall back on similar techniques to provide a slower, but complete variant of our algorithm when our projections cannot reach a conclusion about a decision boundary.

Original: Reluplex (Katz et al., 2017) and its successor, Marabou (Katz et al., 2019) are complete veriﬁcation tools based on SMT solving techniques.
Revised: Reluplex (Katz et al., 2017) and its successor, Marabou (Katz et al., 2019) are complete veriﬁcation tools based on SMT solving techniques.

Original: Unlike our approach, Reluplex and Marabou do not support the (cid:96) 2 norm.
Revised: Unlike our approach, Reluplex and Marabou do not support the (cid:96) 2 norm.

Original: AI2 (Gehr et al., 2018) and its successor, ERAN (Singh et al., 2019b), are based on abstract interpretation (Cousot & Cousot, 1977), using conservative over-approximations to perform their analysis, which leads to false positives, i.e. robust inputs incorrectly classiﬁed as not robust.
Revised: AI2 (Gehr et al., 2018) and its successor, ERAN (Singh et al., 2019b), are based on abstract interpretation (Cousot & Cousot, 1977), using conservative over-approximations to perform their analysis, which leads to false positives, i.e. robust inputs incorrectly classiﬁed as not robust.

Original: A mode of ERAN enables complete veriﬁcation by falling back on a constraint solver when an input is classiﬁed as not robust; however this tool does not support the (cid:96) 2 norm either.
Revised: A mode of ERAN enables complete veriﬁcation by falling back on a constraint solver when an input is classiﬁed as not robust; however this tool does not support the (cid:96) 2 norm either.

Original: FastLin (Weng et al., 2018) exploits the special structure of ReLU networks to efﬁciently compute lower bounds on minimal adversarial distortions.
Revised: FastLin (Weng et al., 2018) exploits the special structure of ReLU networks to efﬁciently compute lower bounds on minimal adversarial distortions.

Original: CROWN (Zhang et al., 2018) later expanded this to general activation functions.
Revised: CROWN (Zhang et al., 2018) later expanded this to general activation functions.

Original: Although FastLin has been shown to be very scalable, our experiments indicate that the computed bounds may be imprecise.
Revised: Although FastLin has been shown to be very scalable, our experiments indicate that the computed bounds may be imprecise.

Original: Recently, a quite different approach has been proposed for robustness certiﬁcation.
Revised: Recently, a quite different approach has been proposed for robustness certiﬁcation.

Original: Randomized
Revised: Randomized

Original: Smoothing (Cohen et al., 2019; Lecuyer et al., 2018) is a post-processing technique that provides a stochastic robustness guarantee at inference time.
Revised: Smoothing (Cohen et al., 2019; Lecuyer et al., 2018) is a post-processing technique that provides a stochastic robustness guarantee at inference time.

Original: This approach differs from our approach in that it (1) modiﬁes the predictions of the original model (increasing the complexity of making predictions), and (2) provides a probabilistic robustness guarantee that is quantiﬁed via a conﬁdence bound.
Revised: This approach differs from our approach in that it (1) modiﬁes the predictions of the original model (increasing the complexity of making predictions), and (2) provides a probabilistic robustness guarantee that is quantiﬁed via a conﬁdence bound.

Original: As such it provides an alternative set of costs and beneﬁts as compared to static veriﬁcation approaches.
Revised: As such it provides an alternative set of costs and beneﬁts as compared to static veriﬁcation approaches.

Original: Its complexity also differs from that of FGP, as it is dependent primarily on the number of samples required to perform its post-processing of the model’s output.
Revised: Its complexity also differs from that of FGP, as it is dependent primarily on the number of samples required to perform its post-processing of the model’s output.

Original: We ﬁnd that in our experimental setup, achieving the same probabilistic guarantee as the experiments described in (Cohen et al., 2019) requires 10 5 samples, taking approximately 4.5 seconds per instance.
Revised: We ﬁnd that in our experimental setup, achieving the same probabilistic guarantee as the experiments described in (Cohen et al., 2019) requires 10 5 samples, taking approximately 4.5 seconds per instance.

Original: Thus, for the models in our evaluation,
Revised: Thus, for the models in our evaluation,

Original: FGP is on average faster or comparable in performance.
Revised: FGP is on average faster or comparable in performance.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we presented a novel approach for verifying the local robustness of networks with piecewise linear activation functions, that relies neither on constraint solving nor conservative overapproximations, but rather on geometric projections.
Revised: In this paper, we presented a novel approach for verifying the local robustness of networks with piecewise linear activation functions, that relies neither on constraint solving nor conservative over-approximations, but rather on geometric projections.

Original: While most existing tools focus on the (cid:96) 1 and (cid:96) ∞ norms, we provide an efﬁcient, highly parallel implementation to certify (cid:96) 2 -robustness.
Revised: While most existing tools focus on the (cid:96)and (cid:96) ∞ norms, we provide an efﬁcient, highly parallel implementation to certify (cid:96) 2 -robustness.

Original: Our implementation outperforms existing exact tools by multiple orders of magnitude, while empirically maintaining the same or better precision under a time constraint.
Revised: Our implementation outperforms existing exact tools by multiple orders of magnitude, while empirically maintaining the same or better precision under a time constraint.

Original: Additionally, we show that our approach is particularly suited to scale up network veriﬁcation to deeper networks—a promising step towards verifying large, state-of-the-art models.
Revised: Additionally, we show that our approach is particularly suited to scale up network veriﬁcation to deeper networks—a promising step towards verifying large, state-of-the-art models.

Original: A C ORRECTNESS P ROOFS A. P ROOF OF T HEOREM
Revised: A C ORRECTNESS P ROOFS A. P ROOF OF T HEOREM

Original: We show that when FGP returns not _ robust , there exists an adversarial example, and when it returns robust , the model is locally robust at x , with radius (cid:15) .
Revised: We show that when FGP returns not _ robust , there exists an adversarial example, and when it returns robust , the model is locally robust at x , with radius (cid:15) .

Original: However, the algorithm may also return unknown , in which case we do not claim anything about the robustness of the model.
Revised: However, the algorithm may also return unknown , in which case we do not claim anything about the robustness of the model.

Original: Proof.
Revised: Proof.

Original: In the ﬁrst case, where FGP returns not _ robust , the proof of Theorem 1 is trivial: we exhibit a point, p , such that (cid:107) x − p (cid:107)≤ (cid:15) , and for which F ( x ) (cid:54) =
Revised: In the ﬁrst case, where FGP returns not _ robust , the proof of Theorem 1 is trivial: we exhibit a point, p , such that (cid:107) x − p (cid:107)≤ (cid:15) , and for which F ( x ) (cid:54) =

Original: F ( p ) .
Revised: F ( p ) .

Original: The interesting case is when FGP returns robust .
Revised: The interesting case is when FGP returns robust .

Original: We prove by contradiction that in this case, x is in fact locally robust with radius (cid:15) .
Revised: We prove by contradiction that in this case, x is in fact locally robust with radius (cid:15) .

Original: Let us assume for the sake of contradiction that FGP returns robust , but there exists a point, p , such that (cid:107) x
Revised: Let us assume for the sake of contradiction that FGP returns robust , but there exists a point, p , such that (cid:107) x

Original: − p (cid:107) ≤ (cid:15) and F ( x ) (cid:54) =
Revised: − p (cid:107) ≤ (cid:15) and F ( x ) (cid:54) =

Original: F ( p ) .
Revised: F ( p ) .

Original: Let ¯ A x
Revised: Let ¯ A x

Original: and ¯ A p be the activation regions associated with x and p respectively.
Revised: and ¯ A p be the activation regions associated with x and p respectively.

Original: We deﬁne a path of activation regions as a sequence, ¯ A 0 ,..., ¯ A k , of activation regions such that the underlying activation patterns A i and A
Revised: We deﬁne a path of activation regions as a sequence, ¯ A 0 ,..., ¯ A k , of activation regions such that the underlying activation patterns A i and A

Original: i +1 differ in exactly one neuron for all i , and there exists at least one input, x i , that has activation pattern
Revised: i +1 differ in exactly one neuron for all i , and there exists at least one input, x i , that has activation pattern

Original: A i for all i .
Revised: A i for all i .

Original: For instance, in a network with three neurons, if A 0 = ( true,true,false ) , A 1 = ( true,false,false ) , and A 2 = ( true,false,true ) , and there exist inputs, x 0 , x 1 , and x 2 with activation patterns A 0 , A 1 , and A 2 , then (cid:104)
Revised: For instance, in a network with three neurons, if A 0 = ( true,true,false ) , A 1 = ( true,false,false ) , and A 2 = ( true,false,true ) , and there exist inputs, x 0 , x 1 , and x 2 with activation patterns A 0 , A 1 , and A 2 , then (cid:104)

Original: ¯ A 0 , ¯ A 1 , ¯ A 2 (cid:105) is a path.
Revised: ¯ A 0 , ¯ A 1 , ¯ A 2 (cid:105) is a path.

Original: Our proof relies on three facts, that we prove hereafter:
Revised: Our proof relies on three facts, that we prove hereafter:

Original: There exists a path, P , from ¯ A x to ¯ A p where each region in the path contains at least one input at distance less than (cid:15) from x , and either ¯ A x =
Revised: There exists a path, P , from ¯ A x to ¯ A p where each region in the path contains at least one input at distance less than (cid:15) from x , and either ¯ A x =

Original: ¯
Revised: ¯

Original: A p or all ¯ A i , ¯ A j in the path are different.
Revised: A p or all ¯ A i , ¯ A j in the path are different.

Original: Our algorithm visits all regions in the path, P . 3.
Revised: Our algorithm visits all regions in the path, P . 3.

Original: If a visited activation region contains an adversarial input, our algorithm either detects it, returning not _ robust , or returns unknown .
Revised: If a visited activation region contains an adversarial input, our algorithm either detects it, returning not _ robust , or returns unknown .

Original: Together, (1) , (2) , and (3) imply that if an adversarial point, p , exists, it resides in an activation region that would have been checked by FGP, which would have resulted in the algorithm returning not _ robust or unknown , contradicting the assumption that it returned robust .
Revised: Together, (1) , (2) , and (3) imply that if an adversarial point, p , exists, it resides in an activation region that would have been checked by FGP, which would have resulted in the algorithm returning not _ robust or unknown , contradicting the assumption that it returned robust .

Original: (1) Existence of a Path Consider the segment going from x to p in a straight line.
Revised: (1) Existence of a Path Consider the segment going from x to p in a straight line.

Original: As (cid:107) x − p (cid:107)≤ (cid:15) , all points on this segment are also at distance less than (cid:15) .
Revised: As (cid:107) x − p (cid:107)≤ (cid:15) , all points on this segment are also at distance less than (cid:15) .

Original: As f is a neural network with ReLU activations, f is a continuous function, as are the activation functions, f u for each of its internal neurons, u .
Revised: As f is a neural network with ReLU activations, f is a continuous function, as are the activation functions, f u for each of its internal neurons, u .

Original: Furthermore, input points at the boundary between two activation regions (i.e., z , such that f u ( z )=0 ) belong to both activation regions.
Revised: Furthermore, input points at the boundary between two activation regions (i.e., z , such that f u ( z )=0 ) belong to both activation regions.

Original: Therefore, listing all activation regions on the segment between x and p yields a path, P , from ¯ A x to ¯ A p , with each region on the path containing an input point, x i , on the segment, such that (cid:107) x − x i (cid:107)≤ (cid:15) .
Revised: Therefore, listing all activation regions on the segment between x and p yields a path, P , from ¯ A x to ¯ A p , with each region on the path containing an input point, x i , on the segment, such that (cid:107) x − x i (cid:107)≤ (cid:15) .

Original: That each ¯ A i , ¯ A j in P is unique follows from the convexity of activation regions: If there exist ¯ A i =
Revised: That each ¯ A i , ¯ A j in P is unique follows from the convexity of activation regions: If there exist ¯ A i =

Original: ¯
Revised: ¯

Original: A j in P such that there exists an ¯ A k in between, then ¯ A i =
Revised: A j in P such that there exists an ¯ A k in between, then ¯ A i =

Original: ¯
Revised: ¯

Original: A j cannot be convex, as there exists a line segment with its end points in ¯ A i and ¯ A j that is not entirely contained within the region ¯
Revised: A j cannot be convex, as there exists a line segment with its end points in ¯ A i and ¯ A j that is not entirely contained within the region ¯

Original: A
Revised: A

Original: i = ¯
Revised: i = ¯

Original: A j .
Revised: A j .

Original: This ensures that paths are of ﬁnite length.
Revised: This ensures that paths are of ﬁnite length.

Original: (2) Exploration by the Algorithm Given the existence of the path, P , from ¯ A x to ¯ A p , we now prove that FGP would visit all activation regions in P if it returns robust .
Revised: (2) Exploration by the Algorithm Given the existence of the path, P , from ¯ A x to ¯ A p , we now prove that FGP would visit all activation regions in P if it returns robust .

Original: We proceed by induction on the length of paths induced (similarly to as above) by a line segment included in the (cid:15) -ball centered on x .
Revised: We proceed by induction on the length of paths induced (similarly to as above) by a line segment included in the (cid:15) -ball centered on x .

Original: In the base case, if the path is of length one, then it contains only ¯ A x , and the claim holds trivially since FGP starts by visiting ¯ A x .
Revised: In the base case, if the path is of length one, then it contains only ¯ A x , and the claim holds trivially since FGP starts by visiting ¯ A x .

Original: In the inductive case, let us assume that for any path with length at most k induced by a segment, s , beginning at x with (cid:107) s (cid:107)≤ (cid:15) , FGP visits all regions in the path.
Revised: In the inductive case, let us assume that for any path with length at most k induced by a segment, s , beginning at x with (cid:107) s (cid:107)≤ (cid:15) , FGP visits all regions in the path.

Original: Now consider a path, P (cid:48)
Revised: Now consider a path, P (cid:48)

Original: =
Revised: =

Original: ¯ A 0 ,..., ¯ A k , of length k +1 , induced by a segment, s (cid:48) , beginning at x with (cid:107) s (cid:48)
Revised: ¯ A 0 ,..., ¯ A k , of length k +1 , induced by a segment, s (cid:48) , beginning at x with (cid:107) s (cid:48)

Original: (cid:107)≤ (cid:15) .
Revised: (cid:107)≤ (cid:15) .

Original: Since ¯ A k − 1 is on the path, there exists a point, x k − 1 , on s (cid:48) such that the sub-segment from x to x k − 1 induces a path of length k ; thus we can apply our induction hypothesis to conclude that FGP visits ¯ A 0 ,..., ¯ A k − 1 .
Revised: Since ¯ A k − 1 is on the path, there exists a point, x k − 1 , on s (cid:48) such that the sub-segment from x to x k − 1 induces a path of length k ; thus we can apply our induction hypothesis to conclude that FGP visits ¯ A 0 ,..., ¯ A k − 1 .

Original: Now, since ¯ A k −and ¯ A k are neighbors in P (cid:48) , they must share some boundary, C , that is intersected by s (cid:48) .
Revised: Now, since ¯ A k − 1 and ¯ A k are neighbors in P (cid:48) , they must share some boundary, C , that is intersected by s (cid:48) .

Original: Thus, since (cid:107) s (cid:48) (cid:107)≤ (cid:15) , d ( x,C ) ≤ (cid:15) ; thus when FGP visits ¯ A k − 1 , it will add ¯ A k to the queue via C .
Revised: Thus, since (cid:107) s (cid:48) (cid:107)≤ (cid:15) , d ( x,C ) ≤ (cid:15) ; thus when FGP visits ¯ A k − 1 , it will add ¯ A k to the queue via C .

Original: Therefore, since FGP returns robust only when all regions in the queue have been visited, FGP will visit ¯ A k , concluding the proof of (2).(3) Detection
Revised: Therefore, since FGP returns robust only when all regions in the queue have been visited, FGP will visit ¯ A k , concluding the proof of (2).

Original:  of Adversarial Examples We conclude by proving that if there exists an adversarial example in a region visited by FGP, then we either return not _ robust or unknown .
Revised: ) Detection of Adversarial Examples We conclude by proving that if there exists an adversarial example in a region visited by FGP, then we either return not _ robust or unknown .

Original: If p in ¯ A p isanadversarialexample, then F ( x ) (cid:54) =
Revised: If p in ¯ A p is an adversarial example, then F ( x ) (cid:54) =

Original: F ( p ) .
Revised: F ( p ) .

Original: Bycontinuityof f , thismeansthatthereexistsan input, p (cid:48) on the segment from x to p that is exactly on a decision boundary, C .
Revised: By continuity of f , this means that there exists an input, p (cid:48) on the segment from x to p that is exactly on a decision boundary, C .

Original: As (cid:107) x − p (cid:48) (cid:107)≤(cid:107) x − p (cid:107)≤ (cid:15) , C must have been analyzed when exploring ¯ A p .
Revised: As (cid:107) x − p (cid:48) (cid:107)≤(cid:107) x − p (cid:107)≤ (cid:15) , C must have been analyzed when exploring ¯ A p .

Original: However, when analyzing a decision boundary, FGP always returns either not _ robust or unknown .
Revised: However, when analyzing a decision boundary, FGP always returns either not _ robust or unknown .

Original: Thus, the decision boundary in the activation region containing p would have been analyzed by FGP; this yields a contradiction, as the algorithm must have returned not _ robust or unknown rather than robust .A. P ROOF OF T HEOREM 1 FOR T REE -B ASED E XPLORATION
Revised: Thus, the decision boundary in the activation region containing p would have been analyzed by FGP; this yields a contradiction, as the algorithm must have returned not _ robust or unknown rather than robust .

Original: We build upon the proof of Theorem 1 presented in Appendix A.1 To prove the correctness of our treebased exploration, we only need to prove that the additional regions we ﬁlter out either are unreachable, or are explored through a different path.

Original: To prove this assertion, we consider the case where we are exploring a region ¯ A , reached from a previously explored region ¯ A prev by ﬂipping a neuron u at layer N .

Original: Let us assume that our optimization ﬁlters out a region ¯

Original: A next reachable from ¯ A by ﬂipping a neuron u (cid:48) .

Original: This means that u (cid:48) belongs to an earlier layer.

Original: As the corresponding activation patterns A and A prev only differ in u , all the constraints corresponding to neurons in earlier layers are the same for ¯ A and ¯ A prev .

Original: In particular, this means that the activation constraint associated to u (cid:48) is the same in both regions.

Original: Hence, ¯ A (cid:48) , where A (cid:48) is A prev with u (cid:48) ﬂipped, is added to the queue of regions to explore.

Original: Now, A (cid:48) and A next only differ in the neuron u , and u is at a later layer than u (cid:48) .

Original: Thus, if ¯ A next is in the (cid:15) -ball, it will be reached from ¯ A (cid:48)

Original: It might be the case that u (cid:48) itself is ﬁltered out in ¯ A prev .

Original: The intuition here is that we can then go "one step up" in the path of regions leading from the initial region to ¯ A and apply a similar argument.

Original: Similarly to the proof of Theorem 1, this can be formalized by an induction on the length of the paths of explored regions.

Original: A. P ROOF OF L OOP I NVARIANT
Revised: A. P ROOF OF L OOP I NVARIANT

Original: The proof of lower-bound variant is similar to the proof of FGP in Section A.1.
Revised: The proof of lower-bound variant is similar to the proof of FGP in Section A.1.

Original: It relies on the following loop invariant.
Revised: It relies on the following loop invariant.

Original: Proof.
Revised: Proof.

Original: Loop Invariant 1 trivially holds on entry of the loop since bound is initialized to 0.
Revised: Loop Invariant 1 trivially holds on entry of the loop since bound is initialized to 0.

Original: Proving that the invariant is maintained is more interesting.
Revised: Proving that the invariant is maintained is more interesting.

Original: Suppose that bound increases to bound (cid:48) .
Revised: Suppose that bound increases to bound (cid:48) .

Original: It must then be shown that there is no unvisited activation region at distance less than bound (cid:48) .
Revised: It must then be shown that there is no unvisited activation region at distance less than bound (cid:48) .

Original: We proceed again by contradiction: assume there was such a region, ¯ A , containing a point, p , such that (cid:107) x − p (cid:107)≤ bound (cid:48) .
Revised: We proceed again by contradiction: assume there was such a region, ¯ A , containing a point, p , such that (cid:107) x − p (cid:107)≤ bound (cid:48) .

Original: Again, let us consider the segment from x to p , and the path, P , it induces.
Revised: Again, let us consider the segment from x to p , and the path, P , it induces.

Original: Let us consider ¯ A i , the ﬁrst region of P at distance greater than bound that was not previously visited.
Revised: Let us consider ¯ A i , the ﬁrst region of P at distance greater than bound that was not previously visited.

Original: If no such region exists, then ¯ A is at distance less than bound from x , and so by our induction hypothesis, it was already visited.
Revised: If no such region exists, then ¯ A is at distancelessthan bound from x , andsobyourinductionhypothesis, itwasalreadyvisited.

Original: otherwise, ¯ A i − 1 was visited, and the activation constraint, C , between ¯ A i − 1 and ¯
Revised: otherwise, ¯ A i −was visited, and the activation constraint, C , between ¯ A i − 1 and ¯

Original: A i is such that d ( x,C ) ≤ bound (cid:48) ≤ (cid:15) .
Revised: A i is such that d ( x,C ) ≤ bound (cid:48) ≤ (cid:15) .

Original: Therefore, C (which leads to ¯ A i ) was already added to the queue with priority less than bound (cid:48) , and by virtue of the priority queue, it was explored before the current iteration, which yields a contradiction.
Revised: Therefore, C (which leads to ¯ A i ) was already added to the queue with priority less than bound (cid:48) , and by virtue of the priority queue, it was explored before the current iteration, which yields a contradiction.

Original: If bound does not increase, the invariant still trivially holds.
Revised: If bound does not increase, the invariant still trivially holds.

Original: This case can happen because our computation of the distance to constraints is an underapproximation of the true distance to the feasible portion of the constraint.[Table]
Revised: This case can happen because our computation of the distance to constraints is an underapproximation of the true distance to the feasible portion of the constraint.

Original: B C OMPUTING A CTIVATION C ONSTRAINTS
Revised: B C OMPUTING A CTIVATION C ONSTRAINTS

Original: Recall that an activation constraint, C u , which is satisﬁed when f u ( x ) ≥ 

Revised: Recall that an activation constraint, C u , which is satisﬁed when f u ( x ) ≥

Original: ⇐⇒
Revised: ⇐⇒

Original: A ( u ) , is a linear constraint with coefﬁcients w u , and intercepts, b u .
Revised: A ( u ) , is a linear constraint with coefﬁcients w u , and intercepts, b u .

Original: The computation of these weights and intercepts does not depend on a particular point in ¯ A — only on the activation pattern, A .
Revised: The computation of these weights and intercepts does not depend on a particular point in ¯ A — only on the activation pattern, A .

Original: Thus, we can compute the boundaries of an activation region, ¯ A , knowing only the corresponding activation pattern, A .
Revised: Thus, we can compute the boundaries of an activation region, ¯ A , knowing only the corresponding activation pattern, A .

Original: In practice, the coefﬁcients, w u , correspond to the gradient of f u with respect to its inputs, evaluated at a point in ¯ A .
Revised: In practice, the coefﬁcients, w u , correspond to the gradient of f u with respect to its inputs, evaluated at a point in ¯ A .

Original: However, frameworks that perform automatic differentiation typically require a concrete point for evaluation of the gradient.
Revised: However, frameworks that perform automatic differentiation typically require a concrete point for evaluation of the gradient.

Original: Thus, we compute the gradient via backpropagation with the activation vectors, a i , where the position in a i corresponding to neuron, u , takes value 1 if A ( u ) is true and 0 otherwise.
Revised: Thus, we compute the gradient via backpropagation with the activation vectors, a i , where the position in a i corresponding to neuron, u , takes value 1 if A ( u ) is true and 0 otherwise.

Original: The intercepts, b u , are computed via a forward computation using the activation vectors, with x set to 0.
Revised: The intercepts, b u , are computed via a forward computation using the activation vectors, with x set to 0.

Original: These operations can easily be implemented to run efﬁciently on a GPU.
Revised: These operations can easily be implemented to run efﬁciently on a GPU.

Original: C A DDITIONAL H EURISTICS FOR FGP

Original: To improve the performance and precision of our algorithm, we use several additional heuristics.

Original: Batching.

Original: The gradient backpropagations for computing constraints and the dot product projections for calculating distances lend themselves well to batching on the GPU.

Original: In our implementation we dequeue multiple elements from the queue at once and calculate the constraints and constraint distances for the corresponding regions in parallel, leading to a speedup of up to 2x on large examples for a well-chosen batch size

Original: ( ∼ 10-100)

Original: Caching Intermediate Computations.

Original: We observe that activation constraints for the ﬁrst layer are invariant across all regions, as there is no earlier neuron to depend on.

Original: Thus, we can precompute all such constraints once for all inputs to be analyzed.

Original: Exploring the Full Queue.

Original: If the analysis of a decision boundary is inconclusive, the FGP algorithm, as presented, stops and returns unknown .

Original: We also implement a variant where we instead record the presence of an ambiguous decision boundary but continue exploring other regions in the queue.

Original: If we empty the queue after unsuccessfully analyzing a decision boundary, we nevertheless return unknown .

Original: However, oftentimes a true adversarial example is found during the continued search, allowing us to return not _ robust conclusively.

Original: Table 2 displays the results of experiments evaluating the impact of exploring the entire search queue instead of stopping at the ﬁrst decision boundary (as described in Section 2.1).

Original: The experimental results show that this heuristic decreases the number of unknown results by approximately 50% while only having a minor impact on the speed of the execution.

Original: Moreover, we reduce the number of unknown results to the extent that we recover the results obtained by GeoCert in every instance for which GeoCert terminates, except 3 non-robust points on mnist40x3, while nevertheless performing our analysis two to three orders of magnitude more quickly.

Original: D H YPERPARAMETERS
Revised: C H YPERPARAMETERS

Original: Here, we provide details on the hyperparameters used to train the models used in our evaluation and to conduct the PGD (Madry et al., 2018) attacks used to obtain the upper bounds on the veriﬁed robust accuracy (VRA).
Revised: Here, we provide details on the hyperparameters used to train the models used in our evaluation and to conduct the PGD (Madry et al., 2018) attacks used to obtain the upper bounds on the veriﬁed robust accuracy (VRA).

Original: CNN Architecture.
Revised: CNN Architecture.

Original: We used a CNN architecture that has been used for benchmarking veriﬁcation by (Wong & Kolter, 2018) and (Croce et al., 2019).
Revised: We used a CNN architecture that has been used for benchmarking veriﬁcation by (Wong & Kolter, 2018) and (Croce et al., 2019).

Original: It contains 2 convolutional layers, each using 4 ×ﬁlters with a 2 × 2 stride, with 16 and 32 channels respectively, followed by a dense layer of 100 nodes.
Revised: It contains 2 convolutional layers, each using 4 ×ﬁlters with a 2 × 2 stride, with 16 and 32 channels respectively, followed by a dense layer of 100 nodes.

Original: PGD Adversarially-trained Models.
Revised: PGD Adversarially-trained Models.

Original: To train models for robustness, we used PGD adversarial training (Madry et al., 2018).
Revised: To train models for robustness, we used PGD adversarial training (Madry et al., 2018).

Original: For training, we used the (cid:96) 2 norm, and let (cid:15)
Revised: For training, we used the (cid:96) 2 norm, and let (cid:15)

Original: = 2 . 5 —10 times the (cid:15) we verify with in order to have a higher fraction of veriﬁably-robust points.
Revised: = 2 . 5 —10 times the (cid:15) we verify with in order to have a higher fraction of veriﬁably-robust points.

Original: We trained each model forepochs with a batch size of 128 and 50 PGD steps.
Revised: We trained each model for 20 epochs with a batch size of 128 and 50 PGD steps.

Original: Maximum-Margin-Regularization-trained Models.
Revised: Maximum-Margin-Regularization-trained Models.

Original: To train models for both robustness and veriﬁability, we used maximum margin regularization (Croce et al., 2019)
Revised: To train models for both robustness and veriﬁability, we used maximum margin regularization (Croce et al., 2019)

Original: (MMR).
Revised: (MMR).

Original: MMR has several hyperparameters including γ B and γ D , which correspond to the desired distance of the training points from the internal boundaries and decision boundaries respectively (these can be seen as acting similarly to the choice of (cid:15) in PGD training); n B and n D , which specify the number of internal and decision boundaries to be moved in each update; and λ , which speciﬁes the relative weight of the regularization as opposed to the regular loss.
Revised: MMR has several hyperparameters including γ B and γ D , which correspond to the desired distance of the training points from the internal boundaries and decision boundaries respectively (these can be seen as acting similarly to the choice of (cid:15) in PGD training); n B and n D , which specify the number of internal and decision boundaries to be moved in each update; and λ , which speciﬁes the relative weight of the regularization as opposed to the regular loss.

Original: See (Croce et al., 2019) for more details on these hyperparameters.
Revised: See (Croce et al., 2019) for more details on these hyperparameters.

Original: We set γ B = γ D =2 . 5 , n B =100 , n D =9 , and λ =0 .
Revised: We set γ B = γ D =2 . 5 , n B =100 , n D =9 , and λ =0 .

Original: 5 , and trained each model for 20 epochs with a batch size of 128.
Revised: 5 , and trained each model for 20 epochs with a batch size of 128.

Original: ReLU-Stability-trained Models.
Revised: ReLU-Stability-trained Models.

Original: We also used ReLU stability (RS) (Xiao et al., 2019) to regularize models for veriﬁable robustness.
Revised: We also used ReLU stability (RS) (Xiao et al., 2019) to regularize models for veriﬁable robustness.

Original: We trained using RS with PGD adversarial loss and (cid:96) 1 weight regularization, as was done by Xiao et al..
Revised: We trained using RS with PGD adversarial loss and (cid:96) 1 weight regularization, as was done by Xiao et al..

Original: We weighted the RS loss by α = 2 . 0 , using an (cid:15) (i.e., the distance over which the ReLU activations should remain stable) of 8 / 255 , and weighted the PGD adversarial loss by β =1 . 0 , using an (cid:15) (i.e., the target robustness radius) of 36 / 255 .
Revised: We weighted the RS loss by α =2 . 0 , using an (cid:15) (i.e., the distance over which the ReLU activations should remain stable) of 8 / 255 , and weighted the PGD adversarial loss by β =1 . 0 , using an (cid:15) (i.e., the target robustness radius) of 36 / 255 .

Original: We scheduled the (cid:96) 1 regularization to decay from 10 − 2 to 10 − 3 over the course of training, and trained for 100 epochs with a batch size of 128.
Revised: We scheduled the (cid:96) 1 regularization to decay from 10 − 2 to 10 − 3 over the course of training, and trained for 100 epochs with a batch size of 128.

Original: PGD Attacks.
Revised: PGD Attacks.

Original: We obtained an upper bound on the VRA by performing PGD attacks on all the correctly-classiﬁed points for which every method either timed out or reported unknown ; the upper bound is the best VRA plus the fraction of points that were correctly-classiﬁed, undecided, and for which the PGD attack did not successfully ﬁnd an adversarial example.
Revised: We obtained an upper bound on the VRA by performing PGD attacks on all the correctly-classiﬁed points for which every method either timed out or reported unknown ; the upper bound is the best VRA plus the fraction of points that were correctly-classiﬁed, undecided, and for which the PGD attack did not successfully ﬁnd an adversarial example.

Original: We conducted these attacks with an (cid:96) 2 bound of (cid:15) =0 .
Revised: We conducted these attacks with an (cid:96) 2 bound of (cid:15) =0 .

Original: 25 with 1,000 PGD steps.
Revised: 25 with 1,000 PGD steps.

Original: E E VALUATION FOR THE (cid:96) ∞ NORM

Original: As explained in Section 3.3, our implementation of FGP is highly modular with respect to the norm used.

Original: As such, we added support for (cid:96) ∞ robustness certiﬁcation by providing a projection and a projected distance computation in FGP.

Original: In this section, we compare the efﬁciency of this implementation to GeoCert (Jordan et al., 2019) and

Original: ERAN (Singh et al., 2019a).

Original: ERAN uses abstract interpretation to analyze networks using conservative over-approximations.

Original: As such, the analysis can certify that an input is robust, but can yield false positives when ﬂagging an input as not robust.

Original: In this evaluation, we use the DeepPoly abstract domain, and the complete mode of ERAN which falls back on constraint solving when an input is classiﬁed as not robust.

Original: Our experimental setup is similar to the one presented in Section 3.1.

Original: For each network, we arbitrarily pick100 inputs from the  test set and report the median analysis time of each tool. We give each tool a time budget of 120 seconds per input. We report the number of instances on which each tool terminates within this time frame, as well as the results of the analyses that terminate. Results are given in Tablefor an (cid:96) ∞ norm bound of (cid:15) =0 .
Revised: Arbitrary Test Instances To arbitrarily pick the 100 test instances, we used the random.randint function from Numpy.

Original: 01 , which contains roughly the same volume as the (cid:96) 2 ball used in most of our evaluation.

Original: We refer to each model by the dataset it was trained on, followed by the architecture of the hidden layers.

Original: For example, “mnist20x3” refers to a model trained on MNIST with 3 hidden layers of 20 neurons each.

Original: Models marked with an asterisk were trained with MMR (Croce et al., 2019); other models were trained using PGD adversarial training (Madry et al., 2018).

Original: We observe that we consistently outperform GeoCert by about two orders of magnitude, both on models trained with MMR and using adversarial training.

Original: On models trained for robustness (upper rows [Table] of Table 3) , FGP is 0.5x to 5x faster than ERAN.
Revised: We set the initial seed to 4444.

Original: In particular, FGP seems to scale better to deeper networks, while ERAN performs better on wider networks.

Original: We attribute this to the effectiveness of the heuristic presented in Section 2.2.

Original: Interestingly, ERAN does not beneﬁt as much from MMR training as FGP and GeoCert.

Original: On large models trained with MMR, our tool is one to two orders of magnitude faster than ERAN, and ERAN is only about 2x faster than GeoCert.

Original: Finally, while FGP is able to determine robustness for almost all inputs deemed robust by GeoCert or ERAN, it struggles to ﬁnd adversarial examples for non-robust points, which results in a higher number of unknown cases compared to the (cid:96) 2 variant.

Original: This is not highly surprising, as projections are better-suited to the Euclidean space than to (cid:96) ∞ space.

Original: While the projection in Euclidean space is unique, this is not the case in the (cid:96) ∞ space; we pick one arbitrary projection which thus might not lead to an adversarial example.