Original: B OOSTING R OBUSTNESS C ERTIFICATION OF N EURAL N ETWORKS
Revised: B OOSTING R OBUSTNESS C ERTIFICATION OF N EURAL N ETWORKS

Original: A BSTRACT
Revised: A BSTRACT

Original: We present a novel approach for the certiﬁcation of neural networks against adversarial perturbations which combines scalable overapproximation methods with precise (mixed integer) linear programming.
Revised: We present a novel approach for the certiﬁcation of neural networks against adversarial perturbations which combines scalable overapproximation methods with precise (mixed integer) linear programming.

Original: This results in signiﬁcantly better precision than state-of-the-art veriﬁers on challenging feedforward and convolutional neural networks with piecewise linear activation functions
Revised: This results in signiﬁcantly better precision than state-of-the-art veriﬁers on challenging feedforward and convolutional neural networks with piecewise linear activation functions

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural networks are increasingly applied in critical domains such as autonomous driving (Bojarski et al., 2016), medical diagnosis (Amato et al., 2013), and speech recognition (Hinton et al., 2012).
Revised: Neural networks are increasingly applied in critical domains such as autonomous driving (Bojarski et al., 2016), medical diagnosis (Amato et al., 2013), and speech recognition (Hinton et al., 2012).

Original: However, it has been shown by Goodfellow et al.
Revised: However, it has been shown by Goodfellow et al.

Original: (2014) that neural networks can be vulnerable against adversarial attacks, i.e., imperceptible input perturbations cause neural networks to misclassify.
Revised: (2014) that neural networks can be vulnerable against adversarial attacks, i.e., imperceptible input perturbations cause neural networks to misclassify.

Original: To address this challenge and prove that a network is free of adversarial examples (usually, in a region around a given input), recent work has started investigating the use of certiﬁcation techniques.
Revised: To address this challenge and prove that a network is free of adversarial examples (usually, in a region around a given input), recent work has started investigating the use of certiﬁcation techniques.

Original: Current veriﬁers can be broadly classiﬁed as either complete or incomplete.
Revised: Current veriﬁers can be broadly classiﬁed as either complete or incomplete.

Original: Complete veriﬁers are exact, i.e., if the veriﬁer fails to certify a network then the network is nonrobust (and vice-versa).
Revised: Complete veriﬁers are exact, i.e., if the veriﬁer fails to certify a network then the network is nonrobust (and vice-versa).

Original: Existing complete veriﬁers are based on Mixed Integer Linear Programming (MILP) (Lomuscio & Maganti, 2017; Fischetti & Jo, 2018; Dutta et al., 2018; Cheng et al., 2017) or SMT solvers (Katz et al., 2017; Ehlers, 2017).
Revised: Existing complete veriﬁers are based on Mixed Integer Linear Programming (MILP) (Lomuscio & Maganti, 2017; Fischetti & Jo, 2018; Dutta et al., 2018; Cheng et al., 2017) or SMT solvers (Katz et al., 2017; Ehlers, 2017).

Original: Although precise, these can only handle networks with a small number of layers and neurons.
Revised: Although precise, these can only handle networks with a small number of layers and neurons.

Original: To scale, incomplete veriﬁers usually employ overapproximation methods and hence they are sound but may fail to prove robustness even if it holds.
Revised: To scale, incomplete veriﬁers usually employ overapproximation methods and hence they are sound but may fail to prove robustness even if it holds.

Original: Incomplete veriﬁers use methods such as duality (Dvijotham et al., 2018), abstract interpretation (Gehr et al., 2018; Singh et al., 2018; 2019), linear approximations (Weng et al., 2018; Wong & Kolter, 2018; Zhang et al., 2018), semideﬁnite relaxations (Raghunathan et al., 2018), combination of linear and non-linear approximation (Xiang et al., 2017), or search space discretization (Huang et al., 2017).
Revised: Incomplete veriﬁers use methods such as duality (Dvijotham et al., 2018), abstract interpretation (Gehr et al., 2018; Singh et al., 2018; 2019), linear approximations (Weng et al., 2018; Wong & Kolter, 2018; Zhang et al., 2018), semideﬁnite relaxations (Raghunathan et al., 2018), combination of linear and non-linear approximation (Xiang et al., 2017), or search space discretization (Huang et al., 2017).

Original: Incomplete veriﬁers are more scalable than complete ones, but can suffer from precision loss for deeper networks.
Revised: Incomplete veriﬁers are more scalable than complete ones, but can suffer from precision loss for deeper networks.

Original: In principle, incomplete veriﬁers can be made asymptotically complete by iteratively reﬁning the input space (Wang et al., 2018a;b); however, at the limit, this eliminates any scalability gains and defeats the purpose of using overapproximation in the ﬁrst place.
Revised: In principle, incomplete veriﬁers can be made asymptotically complete by iteratively reﬁning the input space (Wang et al., 2018a;b); however, at the limit, this eliminates any scalability gains and defeats the purpose of using overapproximation in the ﬁrst place.

Original: This work: boosting complete and incomplete veriﬁers.
Revised: This work: boosting complete and incomplete veriﬁers.

Original: A key challenge then is to design a veriﬁer which improves the precision of incomplete methods and the scalability of complete ones.
Revised: A key challenge then is to design a veriﬁer which improves the precision of incomplete methods and the scalability of complete ones.

Original: In this work, we make a step towards addressing this challenge based on two key ideas: (i) a combination of state-of-the-art overapproximation techniques used by incomplete methods, including LP relaxations, together with MILP solvers, often employed in complete veriﬁers; (ii) a novel heuristic, which points to neurons whose approximated bounds should be reﬁned.
Revised: In this work, we make a step towards addressing this challenge based on two key ideas: (i) a combination of state-of-the-art overapproximation techniques used by incomplete methods, including LP relaxations, together with MILP solvers, often employed in complete veriﬁers; (ii) a novel heuristic, which points to neurons whose approximated bounds should be reﬁned.

Original: We implemented these ideas in a system called ReﬁneAI , and showed that is is faster than state-of-the-art complete veriﬁers on small networks while improving precision of existing incomplete veriﬁers on larger networks.
Revised: We implemented these ideas in a system called ReﬁneAI , and showed that is is faster than state-of-the-art complete veriﬁers on small networks while improving precision of existing incomplete veriﬁers on larger networks.

Original: The recent work of Tjeng et al.
Revised: The recent work of Tjeng et al.

Original: (2019) has also explored the combination of linear programming with over-approximation, though it only considers the simple interval based approximation.
Revised: (2019) has also explored the combination of linear programming with over-approximation, though it only considers the simple interval based approximation.

Original: Our evaluation shows that for the complete veriﬁcation of a 3 × 50 network, ReﬁneAI is faster, while for the larger
Revised: Our evaluation shows that for the complete veriﬁcation of a 3 × 50 network, ReﬁneAI is faster, while for the larger

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Main contributions.
Revised: Main contributions.

Original: Our main contributions are:
Revised: Our main contributions are:

Original: • A reﬁnement-based approach for certifying neural network robustness that combines the strengths of fast overapproximation methods with MILP solvers and LP relaxations.
Revised: • A reﬁnement-based approach for certifying neural network robustness that combines the strengths of fast overapproximation methods with MILP solvers and LP relaxations.

Original: • A novel heuristic for selecting neurons whose bounds should be further reﬁned.
Revised: • A novel heuristic for selecting neurons whose bounds should be further reﬁned.

Original: • A complete end-to-end implementation of our approach in a system called ReﬁneAI , publicly available at https://github.com/eth-sri/eran.
Revised: • A complete end-to-end implementation of our approach in a system called ReﬁneAI , publicly available at https://github.com/eth-sri/eran.

Original: • An evaluation, showing that ReﬁneAI is more precise than existing state-of-the-art incomplete veriﬁers on larger networks and faster (while being complete) than complete veriﬁers on smaller networks.
Revised: • An evaluation, showing that ReﬁneAI is more precise than existing state-of-the-art incomplete veriﬁers on larger networks and faster (while being complete) than complete veriﬁers on smaller networks.

Original:  O VERVIEW
Revised:  O VERVIEW

Original: We now demonstrate how our method improves the precision of a state-of-the-art incomplete veriﬁer.
Revised: We now demonstrate how our method improves the precision of a state-of-the-art incomplete veriﬁer.

Original: The main objective here is to provide an intuitive understanding of our approach; full formal details are provided in the next section.
Revised: The main objective here is to provide an intuitive understanding of our approach; full formal details are provided in the next section.

Original: Consider the simple fully connected feed forward neural network with ReLU activations shown in Fig.
Revised: Consider the simple fully connected feed forward neural network with ReLU activations shown in Fig.

Original: There are two inputs to the network, both in the range [0 , 1] .
Revised: There are two inputs to the network, both in the range [0 , 1] .

Original: The network consists of an input layer, two hidden layers, and one output layer.
Revised: The network consists of an input layer, two hidden layers, and one output layer.

Original: Each layer consist of two neurons each.
Revised: Each layer consist of two neurons each.

Original: For our explanation, we separate each neuron into two parts: one represents the output of the afﬁne transformation while the other captures the output of the ReLU activation.
Revised: For our explanation, we separate each neuron into two parts: one represents the output of the afﬁne transformation while the other captures the output of the ReLU activation.

Original: The weights for the afﬁne transformation are represented by weights on the edges.
Revised: The weights for the afﬁne transformation are represented by weights on the edges.

Original: The bias for each node is shown above or below it.
Revised: The bias for each node is shown above or below it.

Original: Our goal is to verify that for any input in [0 , 1] ×
Revised: Our goal is to verify that for any input in [0 , 1] ×

Original: [0 , 1] , the output at neuron x 13 is greater than the output at x 14 .
Revised: [0 , 1] , the output at neuron x 13 is greater than the output at x 14 .

Original: We now demonstrate how our veriﬁer operates on this network.
Revised: We now demonstrate how our veriﬁer operates on this network.

Original: We assume that the analysis results after the second afﬁne transformation are reﬁned using a MILP formulation of the network whereas the results after the third afﬁne transformation are reﬁned by an LP formulation of the network.
Revised: We assume that the analysis results after the second afﬁne transformation are reﬁned using a MILP formulation of the network whereas the results after the third afﬁne transformation are reﬁned by an LP formulation of the network.

Original: In the next section, we will explain our heuristic for selecting MILP or LP formulations of different neurons in the network.
Revised: In the next section, we will explain our heuristic for selecting MILP or LP formulations of different neurons in the network.

Original: Our analysis leverages the Zonotope domain (Ghorbal et al., 2009) together with the abstract Zonotope transformers specialized to neural network activations as used in DeepZ
Revised: Our analysis leverages the Zonotope domain (Ghorbal et al., 2009) together with the abstract Zonotope transformers specialized to neural network activations as used in DeepZ

Original: [Figure]
Revised: [Figure]

Original: (Singh et al., 2018), a state of the art veriﬁer for neural network robustness.
Revised: (Singh et al., 2018), a state of the art veriﬁer for neural network robustness.

Original: The Zonotope domain associates an afﬁne form ˆ x with each neuron x in the network:
Revised: The Zonotope domain associates an afﬁne form ˆ x with each neuron x in the network:

Original: [Equation]
Revised: [Equation]

Original: Here, c 0 , c i ∈ R are real coefﬁcients and η i ∈ [ s i , t i ] ⊆
Revised: Here, c 0 , c i ∈ R are real coefﬁcients and η i ∈ [ s i , t i ] ⊆

Original: [ − 1 , 1] are the noise symbols, which are shared between the afﬁne forms for different neurons.
Revised: [ − 1 , 1] are the noise symbols, which are shared between the afﬁne forms for different neurons.

Original: This sharing makes the domain relational and thus more precise than non-relational domains such as Interval (Box).
Revised: This sharing makes the domain relational and thus more precise than non-relational domains such as Interval (Box).

Original: An abstract element in our analysis is an intersection between a Zonotope (given as a list of afﬁne forms) and a bounding box.
Revised: An abstract element in our analysis is an intersection between a Zonotope (given as a list of afﬁne forms) and a bounding box.

Original: Thus, for each neuron x , we keep the afﬁne form ˆ x and an interval [ l x , u x ] .
Revised: Thus, for each neuron x , we keep the afﬁne form ˆ x and an interval [ l x , u x ] .

Original: First layer.
Revised: First layer.

Original: Our analysis starts by setting
Revised: Our analysis starts by setting

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: representing the input [0 , 1] at x 1 and [0 , 1] at x 2 in our domain, respectively.
Revised: representing the input [0 , 1] at x 1 and [0 , 1] at x 2 in our domain, respectively.

Original: Next, an afﬁne transformation is applied on the inputs resulting in the output
Revised: Next, an afﬁne transformation is applied on the inputs resulting in the output

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Note that the Zonotope afﬁne transformer is exact for this transformation.
Revised: Note that the Zonotope afﬁne transformer is exact for this transformation.

Original: Next, the Zonotope ReLU transformer is applied.
Revised: Next, the Zonotope ReLU transformer is applied.

Original: We note that as l 3 ≥ 0 , the neuron x 3 provably takes only non-negative values.
Revised: We note that as l 3 ≥ 0 , the neuron x 3 provably takes only non-negative values.

Original: Thus, the ReLU Zonotope transformer outputs ˆ x 5 = ˆ x 

Revised: Thus, the ReLU Zonotope transformer outputs ˆ x 5 = ˆ x

Original: and we set l 5 = l 3 , u 5 = l 3 which is the exact result.
Revised: and we set l 5 = l 3 , u 5 = l 3 which is the exact result.

Original: For x 4 , l 4 < 0 and u 4 > 0 and thus neuron x 4 can take both positive and negative values.
Revised: For x 4 , l 4 < 0 and u 4 > 0 and thus neuron x 4 can take both positive and negative values.

Original: The corresponding output does not have a closed afﬁne form and hence the approximation in blue shown in Fig.
Revised: The corresponding output does not have a closed afﬁne form and hence the approximation in blue shown in Fig.

Original: 2 is used to compute the result.
Revised: 2 is used to compute the result.

Original: This approximation minimizes the area of the result in the input-output plane and introduces a new noise symbol η 3 ∈ [ − 1 , 1] .
Revised: This approximation minimizes the area of the result in the input-output plane and introduces a new noise symbol η 3 ∈ [ − 1 , 1] .

Original: The result is
Revised: The result is

Original: [Equation]
Revised: [Equation]

Original: Note that the Zonotope approximation for x 6 from Fig.
Revised: Note that the Zonotope approximation for x 6 from Fig.

Original: 2 permits negative values whereas x 6 can only take non-negative values in the concrete.
Revised: 2 permits negative values whereas x 6 can only take non-negative values in the concrete.

Original: This over-approximation typically accumulates as the analysis progresses deeper into the network, resulting in overall imprecision and failure to prove properties that actually hold.
Revised: This over-approximation typically accumulates as the analysis progresses deeper into the network, resulting in overall imprecision and failure to prove properties that actually hold.

Original: MILP-based reﬁnement at second layer.
Revised: MILP-based reﬁnement at second layer.

Original: Next, the analysis handles the second afﬁne transformation and computes
Revised: Next, the analysis handles the second afﬁne transformation and computes

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Here, x 7 is provably positive, whereas x 8 can take both positive and negative values.
Revised: Here, x 7 is provably positive, whereas x 8 can take both positive and negative values.

Original: Due to the approximation for x 6 , the bounds for x 7 and x 8 are imprecise.
Revised: Due to the approximation for x 6 , the bounds for x 7 and x 8 are imprecise.

Original: Note that the DeepZ ReLU transformer for x 8 applied next will introduce more imprecision and although the ReLU transformer for provably positive inputs such as x 7 does not lose precision with respect to the input, it still propagates the imprecision in the computation of the abstract values for x 7 .
Revised: Note that the DeepZ ReLU transformer for x 8 applied next will introduce more imprecision and although the ReLU transformer for provably positive inputs such as x 7 does not lose precision with respect to the input, it still propagates the imprecision in the computation of the abstract values for x 7 .

Original: Thus, to reduce precision loss, in our method we reﬁne the bounds for both x 7 and x 8 by formulating the network up to (and including) the second afﬁne transformation as a MILP instance based on a formulation from Tjeng et al.
Revised: Thus, to reduce precision loss, in our method we reﬁne the bounds for both x 7 and x 8 by formulating the network up to (and including) the second afﬁne transformation as a MILP instance based on a formulation from Tjeng et al.

Original: and compute bounds for x 7 and x 8 , respectively.
Revised: and compute bounds for x 7 and x 8 , respectively.

Original: The MILP solver improves the lower bounds for x 7 and x 8 to 1 and − 1 , respectively, which then updates the corresponding lower bounds in our abstraction, i.e., l 7 = 1 and l 8 = − 1 .
Revised: The MILP solver improves the lower bounds for x 7 and x 8 to 1 and − 1 , respectively, which then updates the corresponding lower bounds in our abstraction, i.e., l 7 = 1 and l 8 = − 1 .

Original: Next, the ReLU transformer is applied.
Revised: Next, the ReLU transformer is applied.

Original: Since x 7 is provably positive, we get ˆ x 9 =
Revised: Since x 7 is provably positive, we get ˆ x 9 =

Original: ˆ x 7 , l 9 = l 7 , and u 9 = u 7 .
Revised: ˆ x 7 , l 9 = l 7 , and u 9 = u 7 .

Original: We note that x 8 can take both positive and negative values and is therefore approximated.
Revised: We note that x 8 can take both positive and negative values and is therefore approximated.

Original: However, the ReLU transformer now uses the reﬁned bounds instead of the original bounds and thus the approximation shown in green from Fig.
Revised: However, the ReLU transformer now uses the reﬁned bounds instead of the original bounds and thus the approximation shown in green from Fig.

Original: 2 is used.
Revised: 2 is used.

Original: This approximation has smaller area in the input-output plane compared to the blue one and thus reduces the approximation error.
Revised: This approximation has smaller area in the input-output plane compared to the blue one and thus reduces the approximation error.

Original: The result is
Revised: The result is

Original: [Equation]
Revised: [Equation]

Original: LP-based reﬁnement at ﬁnal layer.
Revised: LP-based reﬁnement at ﬁnal layer.

Original: Continuing with the analysis, we now process the ﬁnal afﬁne transformation, which yields
Revised: Continuing with the analysis, we now process the ﬁnal afﬁne transformation, which yields

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Due to the approximations from previous layers, the computed values can be imprecise.
Revised: Due to the approximations from previous layers, the computed values can be imprecise.

Original: We note that as the analysis proceeds deeper into the network, reﬁning bounds with MILP becomes expensive.
Revised: We note that as the analysis proceeds deeper into the network, reﬁning bounds with MILP becomes expensive.

Original: Thus, we reﬁne the bounds by encoding the network up to (and including) the third afﬁne transformation using the faster LP relaxation of the network based on Ehlers (2017) and compute the bounds for x 11 and x 12 , respectively.
Revised: Thus, we reﬁne the bounds by encoding the network up to (and including) the third afﬁne transformation using the faster LP relaxation of the network based on Ehlers (2017) and compute the bounds for x 11 and x 12 , respectively.

Original: This leads to better results for l 11 = 3 . 25 , l 12 = 2 , and u 12 = 3 .
Revised: This leads to better results for l 11 = 3 . 25 , l 12 = 2 , and u 12 = 3 .

Original: As both x 11 and x 12 are provably positive, the subsequent ReLU transformations set ˆ x 13 =
Revised: As both x 11 and x 12 are provably positive, the subsequent ReLU transformations set ˆ x 13 =

Original: ˆ x 11 , l 13 = l 11 , u 13 =
Revised: ˆ x 11 , l 13 = l 11 , u 13 =

Original: u 11 and ˆ x 14 =
Revised: u 11 and ˆ x 14 =

Original: ˆ x 12 , l 14 = l 12 , u 14 = u 12 .
Revised: ˆ x 12 , l 14 = l 12 , u 14 = u 12 .

Original: Proving robustness.
Revised: Proving robustness.

Original: Since the lower bound l 13 for x 13 is greater than the upper bound u 14 for x 14 , our analysis can prove that the given neural network provides the same label for all inputs in [0 , 1] × [0 , 1] and is thus robust.
Revised: Since the lower bound l 13 for x 13 is greater than the upper bound u 14 for x 14 , our analysis can prove that the given neural network provides the same label for all inputs in [0 , 1] × [0 , 1] and is thus robust.

Original: In contrast, DeepZ without our reﬁnement would compute
Revised: In contrast, DeepZ without our reﬁnement would compute

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: As a result, DeepZ fails to prove that x 13 is greater than x 14 , and thus fails to prove robustness.
Revised: As a result, DeepZ fails to prove that x 13 is greater than x 14 , and thus fails to prove robustness.

Original: Generalization to other abstractions.
Revised: Generalization to other abstractions.

Original: We note that our reﬁnement-based approach is not restricted to the Zonotope domain.
Revised: We note that our reﬁnement-based approach is not restricted to the Zonotope domain.

Original: It can be extended for reﬁning the results computed by other abstractions such as Polyhedra (Singh et al., 2017) or the abstraction used in DeepPoly (Singh et al., 2019).
Revised: It can be extended for reﬁning the results computed by other abstractions such as Polyhedra (Singh et al., 2017) or the abstraction used in DeepPoly (Singh et al., 2019).

Original: For example, the ReLU transformer in Singh et al.
Revised: For example, the ReLU transformer in Singh et al.

Original: (2019) also depends on the bounds of input neurons and thus it will beneﬁt from the precise bounds computed using our reﬁnement.
Revised: (2019) also depends on the bounds of input neurons and thus it will beneﬁt from the precise bounds computed using our reﬁnement.

Original: Since
Revised: Since

Original: DeepPoly often produces more precise results than DeepZ , we believe a combination of this work to DeepPoly will further improve veriﬁcation results.
Revised: DeepPoly often produces more precise results than DeepZ , we believe a combination of this work with DeepPoly will further improve veriﬁcation results.

Original:  O UR A PPROACH
Revised:  O UR A PPROACH

Original: We now describe our approach in more formal terms.
Revised: We now describe our approach in more formal terms.

Original: As in the previous section, we will consider afﬁne transformations and ReLU activations as separate layers.
Revised: As in the previous section, we will consider afﬁne transformations and ReLU activations as separate layers.

Original: As illustrated earlier, the key idea will be to combine abstract interpretation with exact and inexact MILP formulation of the network which is then solved, in order to compute more precise results for neuron bounds.
Revised: As illustrated earlier, the key idea will be to combine abstract interpretation with exact and inexact MILP formulation of the network which is then solved, in order to compute more precise results for neuron bounds.

Original: We begin by describing the core ingredients of abstract interpretation.
Revised: We begin by describing the core ingredients of abstract interpretation.

Original: Our approach requires an abstract domain A n over n variables (i.e., some set whose elements can be encoded symbolically) such as Interval, Zonotope, the abstraction in DeepPoly , or Polyhedra.
Revised: Our approach requires an abstract domain A n over n variables (i.e., some set whose elements can be encoded symbolically) such as Interval, Zonotope, the abstraction in DeepPoly , or Polyhedra.

Original: An abstract domain has a bottom element ⊥ ∈
Revised: An abstract domain has a bottom element ⊥ ∈

Original: A n as well as the following components:
Revised: A n as well as the following components:

Original: • A (potentially non-computable) concretization function γ n : A n → P ( R n ) that associates with each abstract element a ∈ A n the set of concrete points from R n that it abstracts.
Revised: • A (potentially non-computable) concretization function γ n : A n → P ( R n ) that associates with each abstract element a ∈ A n the set of concrete points from R n that it abstracts.

Original: We have, γ n ( ⊥ ) = ∅ .
Revised: We have, γ n ( ⊥ ) = ∅ .

Original: • An abstraction function α n : P ( R n )
Revised: • An abstraction function α n : P ( R n )

Original: →
Revised: →

Original: A n that is computable for (cid:81) i
Revised: A n that is computable for (cid:81) i

Original: [ l i , u i ] ∈ P ( R n ) with (cid:89) i
Revised: [ l i , u i ] ∈ P ( R n ) with (cid:81) i [ l i , u i ] ⊆ γ ( α ( (cid:81) i [ l i , u i ])) where (cid:81) i

Original: [ l i , u i ] = { x ∈ R n | l i ≤ x i ≤ u i } .
Revised: [ l i , u i ] = { x ∈ R n | l i ≤ x i ≤ u i } .

Original: • A bounding box function ι n : A n → R n × R n , where γ n ( a ) ⊆
Revised: • A bounding box function ι n : A n → R n × R n , where γ n ( a ) ⊆

Original: (cid:81) i
Revised: (cid:81) i

Original: [ l i , y i ] for ( l , u ) =
Revised: [ l i , y i ] for ( l , u ) =

Original: ι n ( a ) for all a ∈ A n .
Revised: ι n ( a ) for all a ∈ A n .

Original: • A meet operation a (cid:117) L for each a ∈ A n and linear constraints L over n real variables, where { x ∈ γ n ( a ) | L ( x ) } ⊆
Revised: • A meet operation a (cid:117) L for each a ∈ A n and linear constraints L over n real variables, where { x ∈ γ n ( a ) | L ( x ) } ⊆

Original: γ n ( a (cid:117) L ) .
Revised: γ n ( a (cid:117) L ) .

Original: • An afﬁne abstract transformer T # x
Revised: • An afﬁne abstract transformer T # x

Original: (cid:55)→
Revised: (cid:55)→

Original: Ax + b : A m → A n for each transformation of the form ( x (cid:55)→
Revised: Ax + b : A m → A n for each transformation of the form ( x (cid:55)→

Original: Ax + b ): R m → R n , where { Ax + b | x ∈ γ n ( a ) } ⊆
Revised: Ax + b ): R m → R n , where { Ax + b | x ∈ γ n ( a ) } ⊆

Original: [Equation]

Original:  n ( T # x (cid:55)→ Ax + b ( a )) for all a ∈ A m .
Revised: γ n ( T # x (cid:55)→ Ax + b ( a )) for all a ∈ A m .

Original: • A ReLU abstract transformer T # ReLU | (cid:81) i
Revised: • A ReLU abstract transformer T # ReLU | (cid:81) i

Original: [ li,ui ] : A n → A n , where { ReLU ( x ) | x ∈ γ n ( a ) ∩ (cid:89) i
Revised: [ li,ui ] : A n → A n , where { ReLU ( x ) | x ∈ γ n ( a ) ∩ (cid:89) i

Original: [ l i , u i ] } ⊆
Revised: [ l i , u i ] } ⊆

Original: T # ReLU | (cid:81) i [ li,ui ] ( a ) for all abstract elements a ∈ A n and for all lower and upper bounds l , u ∈ R n on input activations of the ReLU operation.
Revised: T # ReLU | (cid:81) i [ li,ui ] ( a ) for all abstract elements a ∈ A n and for all lower and upper bounds l , u ∈ R n on input activations of the ReLU operation.

Original: Veriﬁcation via Abstract interpretation.
Revised: Veriﬁcation via Abstract interpretation.

Original: As ﬁrst shown by Gehr et al.
Revised: As ﬁrst shown by Gehr et al.

Original: (2018), any such abstract domain induces a method for robustness certiﬁcation of neural networks with ReLU activations.
Revised: (2018), any such abstract domain induces a method for robustness certiﬁcation of neural networks with ReLU activations.

Original: For example, assume that we want to certify that a given neural network f : R m → R n considers class i more likely than class j for all inputs ¯ x with || ¯ x − x || ∞ ≤ (cid:15) for a given x and (cid:15) .
Revised: For example, assume that we want to certify that a given neural network f : R m → R n considers class i more likely than class j for all inputs ¯ x with || ¯ x − x || ∞ ≤ (cid:15) for a given x and (cid:15) .

Original: We can ﬁrst use the abstraction function α m to compute a symbolic overapproximation of the set of possible inputs ¯ x , namelya in = α m ( { ¯ x ∈ R m |
Revised: We can ﬁrst use the abstraction function α m to compute a symbolic overapproximation of the set of possible inputs ¯ x , namely

Original: || ¯ x − x || ∞ ≤ (cid:15) } ) .
Revised: [Equation]

Original: Given that the neural network can be written as a composition of afﬁne functions and ReLU layers, we can then propagate the abstract element a in through the corresponding abstract transformers to obtain a symbolic overapproximation a out of the concrete outputs of the neural network.
Revised: Given that the neural network can be written as a composition of afﬁne functions and ReLU layers, we can then propagate the abstract element a in through the corresponding abstract transformers to obtain a symbolic overapproximation a out of the concrete outputs of the neural network.

Original: For example, if the neural network f ( x ) = A (cid:48) · ReLU ( Ax + b ) + b (cid:48) has a single hidden layer with h hidden neurons, we ﬁrst compute a (cid:48)
Revised: For example, if the neural network f ( x ) = A (cid:48) · ReLU ( Ax + b ) + b (cid:48) has a single hidden layer with h hidden neurons, we ﬁrst compute a (cid:48)

Original: =
Revised: =

Original: T # x (cid:55)→ Ax + b ( a in ) , which is a symbolic overapproximation of the input to the ReLU activation function.
Revised: T # x (cid:55)→ Ax + b ( a in ) , which is a symbolic overapproximation of the input to the ReLU activation function.

Original: We then compute ( l , u ) =
Revised: We then compute ( l , u ) =

Original: ι h ( a (cid:48) ) to obtain opposite corners of a bounding box of all possible ReLU input activations, such that we can apply the ReLU abstract transformer:
Revised: ι h ( a (cid:48) ) to obtain opposite corners of a bounding box of all possible ReLU input activations, such that we can apply the ReLU abstract transformer:

Original: [Equation]
Revised: [Equation]

Original: Finally, we apply the afﬁne abstract transformer again to obtain a out = T # x 
Revised: Finally, we apply the afﬁne abstract transformer again to obtain a out = T # x (cid:55)→

Original: [Equation]
Revised: [Equation]

Original:  Using our assumptions, we can conclude that the set γ n ( a out ) contains all output activations that f can possibly produce when given any of the inputs ¯ x .
Revised: ( a [Equation] Using our assumptions, we can conclude that the set γ n ( a out ) contains all output activations that f can possibly produce when given any of the inputs ¯ x .

Original: Therefore, if a out (cid:117) ( x i ≤ x j ) = ⊥ , we have proved the property: for all ¯ x , the neural network considers class i more likely than class j .
Revised: Therefore, if a out (cid:117) ( x i ≤ x j ) = ⊥ , we have proved the property: for all ¯ x , the neural network considers class i more likely than class j .

Original: Incompleteness.
Revised: Incompleteness.

Original: While this approach is sound (i.e., whenever we prove the property, it actually holds), it is incomplete (i.e., we might not prove the property, even if it holds), because the abstract transformers produce a superset of the set of concrete outputs that the corresponding concrete executions produce.
Revised: While this approach is sound (i.e., whenever we prove the property, it actually holds), it is incomplete (i.e., we might not prove the property, even if it holds), because the abstract transformers produce a superset of the set of concrete outputs that the corresponding concrete executions produce.

Original: This can be quite imprecise for deep neural networks, because the overapproximations introduced in each layer accumulate.
Revised: This can be quite imprecise for deep neural networks, because the overapproximations introduced in each layer accumulate.

Original: Reﬁnement.
Revised: Reﬁning the bounds.

Original: To combat spurious overapproximation, we use mixed integer linear programming (MILP) to compute reﬁned lower and upper bounds l (cid:48) , u (cid:48) after applying each afﬁne abstract transformer (except for the ﬁrst layer).
Revised: To combat spurious overapproximation, we use mixed integer linear programming (MILP) to compute reﬁned lower and upper bounds l (cid:48) , u (cid:48) after applying each afﬁne abstract transformer (except for the ﬁrst layer).

Original: We then reﬁne the abstract element using the meet operator of the underlying abstract domain and the linear constraints l (cid:48)
Revised: We then reﬁne the abstract element using the meet operator of the underlying abstract domain and the linear constraints l (cid:48)

Original: i ≤ x i ≤ u (cid:48)
Revised: i ≤ x i ≤ u (cid:48)

Original: i for all input activations i , i.e., we replace the current abstract element a by a (cid:48)
Revised: i for all input activations i , i.e., we replace the current abstract element a by a (cid:48)

Original: = a (cid:117) ( (cid:86) i
Revised: = a (cid:117) ( (cid:86) i

Original: l
Revised: l

Original: (cid:48)
Revised: (cid:48)

Original: i ≤ x i ≤ u (cid:48) i ) , and continue analysis with the reﬁned abstract element.
Revised: i ≤ x i ≤ u (cid:48) i ) , and continue analysis with the reﬁned abstract element.

Original: Further, and importantly, for ReLU, we obtain a  reﬁned abstract transformer using the new(and tighter) lower and upper bounds.
Revised: Importantly, we obtain a more reﬁned abstract transformer for ReLU than the one used in DeepZ by leveraging the new lower and upper bounds.

Original: For example, the ReLU transformer for y := max (0 , x ) used in DeepZ can be deﬁned using the tighter bounds l (cid:48) x , u (cid:48) x for x  as follows:
Revised: That is, by using the tighter bounds l (cid:48) x , u (cid:48) x for x , we deﬁne the ReLU transformer for y := max (0 , x ) as follows:

Original: [Equation]
Revised: [Equation]

Original: Here λ =
Revised: Here λ =

Original: [Equation]
Revised: [Equation]

Original:  ∈ [ − 1 , 1] is a new noise symbol.
Revised: and (cid:15) new ∈ [ − 1 , 1] is a new noise symbol.

Original: The reﬁned ReLU transformer beneﬁts from the improved bounds.
Revised: The reﬁned ReLU transformer beneﬁts from the improved bounds.

Original: For example, when l x < 0 and u x > 0 holds for the original bounds then after reﬁnement:
Revised: For example, when l x < 0 and u x > 0 holds for the original bounds then after reﬁnement:

Original: • if l (cid:48) x > 0 , then the output is  same as the input and no overapproximation is added.
Revised: • if l (cid:48) x > 0 , then the output is the same as the input and no overapproximation is added.

Original: • else if u (cid:48) x ≤ 0 , then the output is exact.
Revised: • else if u (cid:48) x ≤ 0 , then the output is exact.

Original: • otherwise as shown in Fig.
Revised: • otherwise, as shown in Fig.

Original: 2, the approximation with the tighter l (cid:48) x and u (cid:48) x has smaller area in the input-output plane than the original transformer that uses the imprecise l x and u x .
Revised: 2, the approximation with the tighter l (cid:48) x and u (cid:48) x has smaller area in the input-output plane than the original transformer that uses the imprecise l x and u x .

Original:  To enable reﬁnement, we augment each abstract element with variables and constraints. There is one variable for each neuron, and the constraints  fully capture the behaviour of the neural network up to the last layer whose abstract transformer has been executed.
Revised: Obtaining constraints for reﬁnement. To enable reﬁnement with MILP, we need to obtain the constraints which fully capture the (over-approximated) behavior of the neural network up to the last layer whose abstract transformer has been executed.

Original: We write x ( k ) i to denote the variable corresponding to the activation of the i -th neuron in the k -th layer, where the input layer has k = 0 .
Revised: In our encoding, we have one variable for each neuron and we write x ( k ) i to denote the variable corresponding to the activation of the i -th neuron in the k -th layer, where the input layer has k = 0 .

Original: Similarly, we write l ( k ) i and u ( k ) i to denote the best derived lower and upper bounds for this neuron.
Revised: Similarly, we write l ( k ) i and u ( k ) i to denote the best derived lower and upper bounds for this neuron.

Original: From the input layer, we obtainconstraints of the [-form l 0 i ≤ x (0) i ≤ u 
 i ,-] from afﬁne layers, we obtain constraints of the form x ( k ) i
Revised: We obtain the [-form l 0 i ≤ x (0) i ≤ u 
 i ,-]constraints as follows: from input layer : from afﬁne layer :

Original: = (cid:80)
Revised: [Equation]

Original: j a ( k − 1) ij x ( k− 1)
Revised: For each x ( k ) i := max(0 , x ( k

Original: j + b ( k − 1) i and from ReLU layers we obtain constraints of the form x ( k ) i ≤ x ( k − 1) i − l ( k − 1) i · (1 − z r ) , x ( k ) i ≤ u ( k − 1) i · z r , x ( k ) i ≥ x ( k − 1) i , x ( k ) i ≥ 0 , and z r ∈ { 0 , 1 } for each x ( k ) i :=
Revised: [Equation]

Original: max(0 , x ( k − 1) i ) .
Revised: 1) i ) in ReLU layer : [Equation]

Original: MILP.
Revised: MILP.

Original: Let ϕ ( k ) denote the conjunction of all constraints up to and including those from layer k .
Revised: Let ϕ ( k ) denote the conjunction of all constraints up to and including those from layer k .

Original: To obtain the best possible lower and upper bounds for layer k with p neurons, we need to solve the following 2 · p optimization problems:
Revised: To obtain the best possible lower and upper bounds for layer k with p neurons, we need to solve the following 2 · p optimization problems:

Original: [Equation]
Revised: [Equation]

Original: As was shown by Tjeng et al.
Revised: As was shown by Tjeng et al.

Original: (2019), those optimization problems can be encoded exactly as MILPs, such that the bounds can  be computed with off-the-shelf MILP solvers.
Revised: (2019), such optimization problems can be encoded exactly as MILP instances and the bounds can then be computed with off-the-shelf MILP solvers.

Original: Note that the encoding of subsequent layers depends on the bounds computed in previous layers, where tighter bounds produce simpler MILP instances.

Original: LP relaxation.
Revised: LP relaxation.

Original: While not introducing any approximation, unfortunately, current MILP solvers do not scale to larger neural networks.
Revised: While not introducing any approximation, unfortunately, current MILP solvers do not scale to larger neural networks.

Original: It becomes increasingly more expensive to reﬁne bounds with the MILP-based formulation as the analysis proceeds deeper into the network.
Revised: It becomes increasingly more expensive to reﬁne bounds with the MILP-based formulation as the analysis proceeds deeper into the network.

Original: However, for soundness it is not crucial that the produced bounds are the best possible: for example, plain abstract interpretation uses sound bounds produced by the bounding box function ι instead.
Revised: However, for soundness it is not crucial that the produced bounds are the best possible: for example, plain abstract interpretation uses sound bounds produced by the bounding box function ι instead.

Original: Therefore, for deeper layers in the network, we explore the trade-off between precision and scalability by also considering an intermediate method, which is faster than exact MILP, but also more precise than abstract interpretation.
Revised: Therefore, for deeper layers in the network, we explore the trade-off between precision and scalability by also considering an intermediate method, which is faster than exact MILP, but also more precise than abstract interpretation.

Original: We relax the constraints in ϕ ( k ) in the same way as Ehlers (2017) to obtain a set of weaker linear constraints ϕ ( k ) LP .
Revised: We relax the constraints in ϕ ( k ) in the same way as Ehlers (2017) to obtain a set of weaker linear constraints ϕ ( k ) LP .

Original: We then use the solver to solve the relaxed optimization problems that are constrained by ϕ ( k ) LP instead of ϕ ( k ) , producing possibly looser bounds l (cid:48) ( k ) and u (cid:48) ( k ) .
Revised: We then use the solver to solve the relaxed optimization problems that are constrained by ϕ ( k ) LP instead of ϕ ( k ) , producing possibly looser bounds l (cid:48) ( k ) and u (cid:48) ( k ) .

Original: Note that the encoding of subsequent layers depends on the bounds computed in previous layers, where tighter bounds reduce the amount of newly introduced approximation.
Revised: Note that the encoding of subsequent layers depends on the bounds computed in previous layers, where tighter bounds reduce the amount of newly introduced approximation.

Original: Anytime MILP relaxation.
Revised: Anytime MILP relaxation.

Original: MILP solvers usually provide the option to provide an explicit timeout after which the solver must terminate.
Revised: MILP solvers usually provide the option to provide an explicit timeout after which the solver must terminate.

Original: In return, the solver may not be able to solve the instance exactly, but it will instead provide lower and upper bounds on the objective function in a best-effort fashion.
Revised: In return, the solver may not be able to solve the instance exactly, but it will instead provide lower and upper bounds on the objective function in a best-effort fashion.

Original: This provides another way to compute sound but inexact bounds l (cid:48) ( k ) and u (cid:48) ( k ) .
Revised: This provides another way to compute sound but inexact bounds l (cid:48) ( k ) and u (cid:48) ( k ) .

Original: In practice, we choose a fraction θ ∈ (0 , 1] of neurons in a given layer k and compute bounds for them using MILP with a timeout T in a ﬁrst step.
Revised: In practice, we choose a fraction θ ∈ (0 , 1] of neurons in a given layer k and compute bounds for them using MILP with a timeout T in a ﬁrst step.

Original: In the second step, for a fraction δ ∈ [0 , 1 − θ ] of neurons in the layer, we set the timeout to β · T , where T is the average time taken by the MILP solver to solve one of the instances from the ﬁrst step and β ∈ [0 , 1] is a parameter.
Revised: In the second step, for a fraction δ ∈ [0 , 1 − θ ] of neurons in the layer, we set the timeout to β · T , where T is the average time taken by the MILP solver to solve one of the instances from the ﬁrst step and β ∈ [0 , 1] is a parameter.

Original: Neuron selection heuristic.
Revised: Neuron selection heuristic.

Original: To select the θ -fraction of neurons for the ﬁrst step of the anytime
Revised: To select the θ -fraction of neurons for the ﬁrst step of the anytime

Original: MILP relaxation for the k -th layer, we rank the neurons.
Revised: MILP relaxation for the k -th layer, we rank the neurons.

Original: If the next layer is a ReLU layer, we ﬁrst ignore all neurons whose activations can be proven to be non-positive using abstract interpretation (i.e., using the bounds produced by ι ), because in this case it is already known that ReLU will map the activation to 0 .
Revised: If the next layer is a ReLU layer, we ﬁrst ignore all neurons whose activations can be proven to be non-positive using abstract interpretation (i.e., using the bounds produced by ι ), because in this case it is already known that ReLU will map the activation to 0 .

Original: The remaining neurons are ordered in up to two different ways, once by width (i.e. neuron i has key u ( k ) i − l ( k ) i ), and possibly once by the sum of absolute output weights.
Revised: The remaining neurons are ordered in up to two different ways, once by width (i.e. neuron i has key u ( k ) i − l ( k ) i ), and possibly once by the sum of absolute output weights.

Original: i.e., if the next layer is a fully connected layer x
Revised: i.e., if the next layer is a fully connected layer x

Original: (cid:55)→ Ax + b , the key of neuron i is (cid:80)
Revised: (cid:55)→ Ax + b , the key of neuron i is (cid:80)

Original: j
Revised: j

Original: | A
Revised: | A

Original: i,j | .
Revised: i,j | .

Original: If the next layer is a ReLU layer, we skip the ReLU layer and use the weights from the fully connected layer that follows it (if any).
Revised: If the next layer is a ReLU layer, we skip the ReLU layer and use the weights from the fully connected layer that follows it (if any).

Original: The two ranks of a neuron in both orders are added, and the θ -fraction with smallest rank sum is selected and their bounds are reﬁned with a timeout of T whereas the next δ -fraction of neurons are reﬁned with a timeout of β · T .
Revised: The two ranks of a neuron in both orders are added, and the θ -fraction with smallest rank sum is selected and their bounds are reﬁned with a timeout of T whereas the next δ -fraction of neurons are reﬁned with a timeout of β · T .

Original: ReﬁneAI : end-to-end approach.
Revised: ReﬁneAI : end-to-end approach.

Original: To certify robustness of deep neural networks, we combine
Revised: To certify robustness of deep neural networks, we combine

Original: MILP, LP relaxation, and abstract interpretation.
Revised: MILP, LP relaxation, and abstract interpretation.

Original: We ﬁrst pick numbers of layers k MILP , k LP , k AI that sum to the total number of layers of the neural network.
Revised: We ﬁrst pick numbers of layers k MILP , k LP , k AI that sum to the total number of layers of the neural network.

Original: For the analysis of the k MILP layers, we reﬁne bounds using anytime MILP relaxation with the neuron selection heuristic.
Revised: For the analysis of the k MILP layers, we reﬁne bounds using anytime MILP relaxation with the neuron selection heuristic.

Original: As an optimization, we do not perform reﬁnement after the abstract transformer for the ﬁrst layer in case it is an afﬁne transformation as the abstract domain computes the tightest possible bounding box for an afﬁne transformation of a box (this is always the case in our experiments).
Revised: As an optimization, we do not perform reﬁnement after the abstract transformer for the ﬁrst layer in case it is an afﬁne transformation as the abstract domain computes the tightest possible bounding box for an afﬁne transformation of a box (this is always the case in our experiments).

Original: For the k LP layers, we reﬁne bounds using LP relaxation (i.e., the network up to the layer to be reﬁned is encoded using linear constraints) combined with the neuron selection heuristic.
Revised: For the k LP layers, we reﬁne bounds using LP relaxation (i.e., the network up to the layer to be reﬁned is encoded using linear constraints) combined with the neuron selection heuristic.

Original: For the remaining k AI layers, we use abstract interpretation without additional reﬁnement (however, this also beneﬁts from reﬁnement that was performed in previous layers), and compute the bounds using ι .
Revised: For the remaining k AI layers, we use abstract interpretation without additional reﬁnement (however, this also beneﬁts from reﬁnement that was performed in previous layers), and compute the bounds using ι .

Original: Final property certiﬁcation.
Revised: Final property certiﬁcation.

Original: Let k be the index of the last layer and p be the number of output classes.
Revised: Let k be the index of the last layer and p be the number of output classes.

Original: We can encode the ﬁnal certiﬁcation problem using the output abstract element a out obtained after applying the abstract transformer for the last layer in the network.
Revised: We can encode the ﬁnal certiﬁcation problem using the output abstract element a out obtained after applying the abstract transformer for the last layer in the network.

Original: If we want to prove that class i is assigned a higher probability than class j , it sufﬁces to show that a out (cid:117) ( x ( k ) i ≤ x ( k ) j ) = ⊥ .
Revised: If we want to prove that class i is assigned a higher probability than class j , it sufﬁces to show that a out (cid:117) ( x ( k ) i ≤ x ( k ) j ) = ⊥ .

Original: If this fails, one can resort to complete veriﬁcation using MILP: the property is satisﬁed if and only if the set of constraints ϕ ( k ) ( x (0) 1 , . . .
Revised: If this fails, one can resort to complete veriﬁcation using MILP: the property is satisﬁed if and only if the set of constraints ϕ ( k ) ( x (0) 1 , . . .

Original: , x ( k ) p ) ∧ ( x ( k ) i ≤ x ( k ) j ) is unsatisﬁable.
Revised: , x ( k ) p ) ∧ ( x ( k ) i ≤ x ( k ) j ) is unsatisﬁable.

Original: [Table]
Revised: [Table]

Original:  E VALUATION
Revised:  E VALUATION

Original: We now evaluate the effectiveness of our approach for the problem of robustness veriﬁcation of
Revised: We now evaluate the effectiveness of our approach on the problem of robustness veriﬁcation of

Original: ReLU-based feedforward and convolutional neural networks.
Revised: ReLU-based feedforward and convolutional neural networks.

Original: Our results show that our approach enables faster complete veriﬁcation than the state-of-the-art complete veriﬁer from Tjeng et al.
Revised: The results show that our approach enables faster complete veriﬁcation than the state-of-the-art complete veriﬁer of Tjeng et al.

Original: and produces more precise results than state-of-the-art incomplete veriﬁers:
Revised: and produces more precise results than state-of-the-art incomplete veriﬁers:

Original: DeepZ (Singh et al., 2018) and DeepPoly (Singh et al., 2019), when complete certiﬁcation becomes infeasible.
Revised: DeepZ (Singh et al., 2018) and DeepPoly (Singh et al., 2019), when complete certiﬁcation becomes infeasible.

Original: We implemented our approach inthe form of a Python analyzer which we call ReﬁneAI .
Revised: We implemented our approach in a system called ReﬁneAI .

Original: ReﬁneAI usesthe Zonotope transformers from DeepZ for performing abstract interpretation over the networks and Gurobi (Gurobi Optimization, LLC, 2018) for solving MILP and LP instances. ReﬁneAI builds on the Zonotope and Interval abstractions implemented in the ELINA library (eli, 2018; Singh et al., 2017).
Revised: ReﬁneAI uses Gurobi (Gurobi Optimization, LLC, 2018) for solving MILP and LP instances and is built on top of the ELINA library (eli, 2018; Singh et al., 2017) for numerical program analysis.

Original: All of our code, neural networks, and images used in our experiments are publicly available at https://github.com/eth-sri/eran.
Revised: All of our code, neural networks, and images used in our experiments are publicly available at https://github.com/eth-sri/eran.

Original: Evaluation datasets.
Revised: Evaluation datasets.

Original: We used the popular MNIST (Lecun et al., 1998) and CIFAR10 (Krizhevsky, 2009) datasets in our experiments.
Revised: We used the popular MNIST (Lecun et al., 1998) and CIFAR10 (Krizhevsky, 2009) datasets in our experiments.

Original: Both datasets include a training set of 50 000 and a testing set of 10 000 images.
Revised: Both datasets include a training set of 50 000 and a testing set of 10 000 images.

Original: All images in the MNIST dataset are grayscale images of size 28 × 28 pixels whereas those in CIFAR10 are RGB images of size 32 × 32 .
Revised: All images in the MNIST dataset are grayscale images of size 28 × 28 pixels whereas those in CIFAR10 are RGB images of size 32 × 32 .

Original: Neural networks.
Revised: Neural networks.

Original: Table 1 shows 11 different MNIST and CIFAR10 feedforward (FNNs) and convolutional networks (CNNs) with ReLU activations used in our experiments.
Revised: Table 1 shows 11 different MNIST and CIFAR10 feedforward (FNNs) and convolutional networks (CNNs) with ReLU activations used in our experiments.

Original: Out of thesewere trained to be robust against adversarial attacks using DiffAI (Mirman et al., 2018) whereas the remaining 7 did not have any adversarial training.
Revised: Out of thesewere trained to be robust against adversarial attacks using DiffAI (Mirman et al., 2018) whereas the remaining 7 did not use any adversarial training.

Original: The largest network in our experiments contains > 88 K neurons whereas the deepest network contains 9 layers.
Revised: The largest network in our experiments contains > 88 K neurons whereas the deepest network contains 9 layers.

Original: Robustness properties.
Revised: Robustness properties.

Original: We consider the adversarial region deﬁned by the L ∞ -norm (Carlini &
Revised: We consider the adversarial region deﬁned by the L ∞ -norm (Carlini &

Original: Wagner, 2017) perturbation.
Revised: Wagner, 2017) perturbation.

Original: The region is parameterized by (cid:15) ∈ R and contains all images in which each pixel x i has distance ≤ (cid:15) from the corresponding pixel x i in the original input image x .
Revised: The region is parameterized by (cid:15) ∈ R and contains all images in which each pixel x i has distance ≤ (cid:15) from the corresponding pixel x i in the original input image x .

Original: Our goal is to certify that the neural network produces the same label as on the original image on all images in the adversarial region.
Revised: Our goal is to certify that the neural network produces the same label as on the original image on all points in the adversarial region.

Original: Thus, the adversarial attack considered here is untargeted.
Revised: Thus, the adversarial attack considered here is untargeted.

Original: Experimental setup.
Revised: Experimental setup.

Original: All experiments for the 3 × 50 FNN and all CNNs were carried out on a 2.
Revised: All experiments for the 3 × 50 FNN and all CNNs were carried out on a 2.

Original: GHz 14 core Intel Xeon CPU E5-2690 with 512 GB of main memory; the remaining FNNs were evaluated on a 3.3 GHz 10 Core Intel i9-7900X Skylake CPU with a main memory of 64 GB.
Revised: GHz 14 core Intel Xeon CPU E5-2690 with 512 GB of main memory; the remaining FNNs were evaluated on a 3.3 GHz 10 Core Intel i9-7900X Skylake CPU with a main memory of 64 GB.

Original: Benchmarks.
Revised: Benchmarks.

Original: We selected the ﬁrst 100 images from the test set of both MNIST and CIFAR10.
Revised: We selected the ﬁrst 100 images from the test set of both MNIST and CIFAR10.

Original: Any image on which the neural network misclassiﬁed without perturbation was not considered for robustness certiﬁcation in our experiments.
Revised: Any image on which the neural network misclassiﬁed without perturbation was not considered for robustness certiﬁcation in our experiments.

Original: We consider complete certiﬁcation with ReﬁneAI on the 3 × 50 network and choose an (cid:15) for which the incomplete veriﬁer DeepZ certiﬁed <
Revised: We consider complete certiﬁcation with ReﬁneAI on the 3 × 50 network and choose an (cid:15) for which the incomplete veriﬁer DeepZ certiﬁed <

Original: 40% of all[Table] candidate images.
Revised: 40% of all candidate images.

Original: We consider incomplete certiﬁcation for the remaining networks and choose an (cid:15) for which complete certiﬁcation with ReﬁneAI becomes infeasible. 
Revised: We consider incomplete certiﬁcation for the remaining networks and choose an (cid:15) for which complete certiﬁcation with ReﬁneAI becomes infeasible. [Table]

Original: 4. C OMPLETE CERTIFICATION
Revised: 4. C OMPLETE CERTIFICATION

Original: We ﬁrst consider the exact robustness certiﬁcation of the MNIST 3 × 50 networks against L ∞ -norm attack with (cid:15) = 0 .
Revised: We ﬁrst consider the exact robustness certiﬁcation of the MNIST 3 × 50 networks against L ∞ -norm attack with (cid:15) = 0 .

Original: DeepZ veriﬁes the network robust on 37% of images in the test set for this value of (cid:15) .
Revised: DeepZ veriﬁes the network robust on 37% of images in the test set for this value of (cid:15) .

Original: ReﬁneAI
Revised: ReﬁneAI

Original: ﬁrst runs DeepZ analysis on the whole network collecting the bounds for all neurons in the network.
Revised: ﬁrst runs DeepZ analysis on the whole network collecting the bounds for all neurons in the network.

Original: If DeepZ fails to certify, then the collected bounds are used to encode the robustness certiﬁcation as a MILP instance (discussed in section 3).
Revised: If DeepZ fails to certify, then the collected bounds are used to encode the robustness certiﬁcation as a MILP instance (discussed in section 3).

Original: We compare ReﬁneAI against the approach from Tjeng et al.
Revised: We compare ReﬁneAI against the approach from Tjeng et al.

Original: (2019) which is a state-of-the-art complete veriﬁer.
Revised: (2019) which is a state-of-the-art complete veriﬁer.

Original: This approach is also MILP-based like ours, but it uses Interval analysis and LP to determine neuron bounds.
Revised: This approach is also MILP-based like ours, but it uses Interval analysis and LP to determine neuron bounds.

Original: We implemented the Interval analysis and LP-based analysis to determine the initial bounds.
Revised: We implemented the Interval analysis and LP-based analysis to determine the initial bounds.

Original: We call the
Revised: We call the

Original: MILP solver only if LP analysis (or Interval analysis) fails to certify.
Revised: MILP solver only if LP analysis (or Interval analysis) fails to certify.

Original: All complete veriﬁers certify the neural network to be robust against L ∞ -norm perturbations for 85% of the images.
Revised: All complete veriﬁers certify the neural network to be robust against L ∞ -norm perturbations for 85% of the images.

Original: The average runtimes of ReﬁneAI , MILP with presolve bounds from Interval analysis, and MILP with bounds from LP analysis are 28 , 123 , and 35 seconds respectively.
Revised: The average runtimes of ReﬁneAI , MILP with presolve bounds from Interval analysis, and MILP with bounds from LP analysis are 28 , 123 , and 35 seconds respectively.

Original: Based on our result, we believe that the
Revised: Based on our result, we believe that the

Original: Zonotope analysis offers a good middle ground between the speed of the Interval analysis and the precision of LP for bound computation as it produces precise bounds faster than LP.
Revised: Zonotope analysis offers a good middle ground between the speed of the Interval analysis and the precision of LP for bound computation as it produces precise bounds faster than LP.

Original: 4. I NCOMPLETE CERTIFICATION
Revised: 4. I NCOMPLETE CERTIFICATION

Original: We next consider incomplete certiﬁcation for the remaining networks and compare against DeepZ and DeepPoly .
Revised: We next consider incomplete certiﬁcation for the remaining networks and compare against DeepZ and DeepPoly .

Original: We measure the average runtime per image in seconds and the precision measured by the % of images for which the veriﬁer certiﬁed the network to be robust.
Revised: We measure the average runtime per image in seconds and the precision measured by the % of images for which the veriﬁer certiﬁed the network to be robust.

Original: We note that DeepZ has the same precision as Fast-Lin (Weng et al., 2018) and DeepPoly has the same precision as CROWN (Zhang et al., 2018).
Revised: We note that DeepZ has the same precision as Fast-Lin (Weng et al., 2018) and DeepPoly has the same precision as CROWN (Zhang et al., 2018).

Original: The (cid:15) values used for deﬁning the adversarial regions are shown in Table 2.
Revised: The (cid:15) values used for deﬁning the adversarial regions are shown in Table 2.

Original: The (cid:15) values for networks trained to be robust are larger than for networks that are not.
Revised: The (cid:15) values for networks trained to be robust are larger than for networks that are not.

Original: We note that running the Interval analysis to obtain initial bounds is too imprecise for these large networks with the (cid:15) values considered in our experiments.
Revised: We note that running the Interval analysis to obtain initial bounds is too imprecise for these large networks with the (cid:15) values considered in our experiments.

Original: As a result, the approach from Tjeng et al.
Revised: As a result, the approach from Tjeng et al.

Original: (2019) has to rely on applying LP per neuron to obtain precise bounds for the MILP solver which does not scale.
Revised: (2019) has to rely on applying LP per neuron to obtain precise bounds for the MILP solver which does not scale.

Original: For example, on the 9 × 200 network, determining bounds with LP already takes >minutes (without calling the MILP solver which is more expensive than LP) whereas ReﬁneAI has an average running time of ≈ 14 minutes.
Revised: For example, on the 9 × 200 network, determining bounds with LP already takes >minutes (without calling the MILP solver which is more expensive than LP) whereas ReﬁneAI has an average running time of ≈ 14 minutes.

Original: Parameter values.
Revised: Parameter values.

Original: We experimented with different values of the analysis parameters k MILP , k LP , k AI , θ, δ, β, T and chose those values that offered the best tradeoff between performance and precision for the robustness certiﬁcation for each neural network.
Revised: We experimented with different values of the analysis parameters k MILP , k LP , k AI , θ, δ, β, T and chose those values that offered the best tradeoff between performance and precision for the robustness certiﬁcation for each neural network.

Original: We reﬁne the output of all afﬁne transformations in the network that are followed by a ReLU except the ﬁrst one.
Revised: We reﬁne the output of all afﬁne transformations in the network that are followed by a ReLU except the ﬁrst one.

Original: In a given layer, we consider all neurons ω that can take positive values after applying the afﬁne transformation as candidates for reﬁnement.
Revised: In a given layer, we consider all neurons ω that can take positive values after applying the afﬁne transformation as candidates for reﬁnement.

Original: For the MNIST 5 × 100 , 6 × 100 , 9 × 100 , 6 × 200 , and 9 × 200 FNNs, we reﬁne the candidate neurons in the output of the afﬁne transformations in layers 2 - 4 with MILP and those in the remaining layers using LP.
Revised: For the MNIST 5 × 100 , 6 × 100 , 9 × 100 , 6 × 200 , and 9 × 200 FNNs, we reﬁne the candidate neurons in the output of the afﬁne transformations in layers 2 - 4 with MILP and those in the remaining layers using LP.

Original: For MILP based reﬁnement, we use θ = ω 5 k − 2 · p in layer k with p neurons.
Revised: For MILP based reﬁnement, we use θ = ω 5 k − 2 · p in layer k with p neurons.

Original: For LP based reﬁnement, we use θ = ω 2 k − 5 · p .
Revised: For LP based reﬁnement, we use θ = ω 2 k − 5 · p .

Original: We use timeout T = 1 second, β = 0 . 5 , and δ = ωp − θ for both
Revised: We use timeout T = 1 second, β = 0 . 5 , and δ = ωp − θ for both

Original: MILP and LP based reﬁnements.
Revised: MILP and LP based reﬁnements.

Original: For the CIFAR10 6 × 100 FNN, we use the same values except that we use θ = ω 2 k − 2 · p for MILP reﬁnement and set T = 6 seconds for both MILP and LP based reﬁnement as it is more expensive to reﬁne the neurons in CIFAR10 networks because of more input neurons.
Revised: For the CIFAR10 6 × 100 FNN, we use the same values except that we use θ = ω 2 k − 2 · p for MILP reﬁnement and set T = 6 seconds for both MILP and LP based reﬁnement as it is more expensive to reﬁne the neurons in CIFAR10 networks because of more input neurons.

Original: For the CNNs, we do not reﬁne convolutional layers as the number of candidates in these layers are very large and instead reﬁne all candidate neurons in fully connected layers with  larger timeout  to compensate for  more difﬁcult problem instances for the solver.
Revised: For the CNNs, we do not reﬁne convolutional layers as the number of candidates in these layers is very large and instead reﬁne all candidate neurons in fully connected layers with a larger timeout so to compensate for the more difﬁcult problem instances for the solver.

Original: For the MNIST ConvSmall, ConvBig and CIFAR10 ConvSmall networks, we reﬁne all the candidate neurons with MILP using a timeout  T = 10 seconds.
Revised: For the MNIST ConvSmall, ConvBig and CIFAR10 ConvSmall networks, we reﬁne all the candidate neurons with MILP using a timeout of T = 10 seconds.

Original: For the MNIST ConvSuper network, we reﬁne similarly but use LP instead of MILP with a timeout of T = 15 seconds.
Revised: For the MNIST ConvSuper network, we reﬁne similarly but use LP instead of MILP with a timeout of T = 15 seconds.

Original: Results for incomplete certiﬁcation.
Revised: Results for incomplete certiﬁcation.

Original: Table 2 shows the precision and the average runtime of all three veriﬁers.
Revised: Table 2 shows the precision and the average runtime of all three veriﬁers.

Original: It can be seen that ReﬁneAI either improves or achieves the precision of the state-ofthe-art veriﬁers on all neural networks.
Revised: It can be seen that ReﬁneAI either improves or achieves the precision of the stateof-the-art veriﬁers on all neural networks.

Original: It certiﬁes more images than DeepZ on all networks except the MNIST ConvSuper network.
Revised: It certiﬁes more images than DeepZ on all networks except the MNIST ConvSuper network.

Original: This is becausethat for the (cid:15) considered, DeepZ is already very precise.
Revised: This is because for the (cid:15) considered, DeepZ is already very precise.

Original: We could not try larger (cid:15) for this network as the DeepZ analysis becomes too expensive on this network.
Revised: We could not try larger (cid:15) for this network as the DeepZ analysis becomes too expensive on this network.

Original: ReﬁneAI certiﬁes  network to be more robust on more images than DeepPoly on 6 out of 10 networks.
Revised: ReﬁneAI certiﬁes the network to be more robust on more images than DeepPoly onout of 10 networks.

Original: It can be seen that the number of neurons in the network is not the determining factor for the average runtime of ReﬁneAI .
Revised: It can be seen that the number of neurons in the network is not the determining factor for the average runtime of ReﬁneAI .

Original: We observe that ReﬁneAI runs faster on the networks trained to be robust and the top three networks with the largest runtime for ReﬁneAI are all networks not trained to be robust.
Revised: We observe that ReﬁneAI runs faster on the networks trained to be robust and the top three networks with the largest runtime for ReﬁneAI are all networks not trained to be robust.

Original: This is because robust networks are relatively easier to certify and produce only a small number of candidate neurons for reﬁnement, which are easier to reﬁne by the solver.
Revised: This is because robust networks are relatively easier to certify and produce only a small number of candidate neurons for reﬁnement, which are easier to reﬁne by the solver.

Original: For example, even though the same parameter values are used for reﬁning the results on the MNIST ConvSmall and ConvBig networks, the average runtime of ReﬁneAI on the robustly trained ConvBig network with around 35 K neurons, 6 layers and a perturbation region deﬁned using (cid:15) = 0 .
Revised: For example, even though the same parameter values are used for reﬁning the results on the MNIST ConvSmall and ConvBig networks, the average runtime of ReﬁneAI on the robustly trained ConvBig network with around 35 K neurons, 6 layers and a perturbation region deﬁned using (cid:15) = 0 .

Original: 2 is almost 4 times less than on the non-robust ConvSmall network with only 3 604 neurons, 3 layers and a smaller (cid:15) = 0 .
Revised: 2 is almost 4 times less than on the non-robust ConvSmall network with only 3 604 neurons, 3 layers and a smaller (cid:15) = 0 .

Original: 4. E FFECT OF NEURON SELECTION HEURISTIC
Revised: 4. E FFECT OF NEURON SELECTION HEURISTIC

Original: We use the neuron selection heuristic from section 3 to determine neurons which need to be reﬁned more than others for FNNs as reﬁning all neurons in a layer with MILP can signiﬁcantly slow down the analysis.
Revised: We use the neuron selection heuristic from section 3 to determine neurons which need to be reﬁned more than others for FNNs as reﬁning all neurons in a layer with MILP can signiﬁcantly slow down the analysis.

Original: To check whether our heuristic is able to identify important neurons, we ran the analysis on the MNIST 9 × 200 FNN by keeping all analysis parameters the same, except instead of selecting the neurons with the smallest rank sum ﬁrst we selected the neurons with the largest rank sum ﬁrst (thus reﬁning neurons more if our heuristic deeps them  unimportant).
Revised: To check whether our heuristic is able to identify important neurons, we ran the analysis on the MNIST 9 × 200 FNN by keeping all analysis parameters the same, except instead of selecting the neurons with the smallest rank sum ﬁrst we selected the neurons with the largest rank sum ﬁrst (thus reﬁning neurons more if our heuristic deems them as unimportant).

Original: With this change, the average runtime does not change signiﬁcantly.
Revised: With this change, the average runtime does not change signiﬁcantly.

Original: However, the modiﬁed analysis loses precision and fails to certify two images that the analysis reﬁning with our neuron selection heuristic certiﬁes.
Revised: However, the modiﬁed analysis loses precision and fails to certify two images that the analysis reﬁning with our neuron selection heuristic succeeds on.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we presented a  reﬁnement-based approach for increasing the precision of incomplete veriﬁers for feedforward and convolutional neural networks with ReLU activations.
Revised: In this paper we presented a novel reﬁnement-based approach for effectively combining overapproximation techniques used by incomplete veriﬁers together with linear programming based methods used in complete veriﬁers. We implemented our method in a system called ReﬁneAI and showed its effectiveness on veriﬁcation tasks involving feedforward and convolutional neural networks with ReLU activations.

Original: This enabled us to certify robustness properties that existing state-of-the-art complete veriﬁers cannot handle (due to scalability issues) whereas state-of-the-art incomplete veriﬁers could not prove as they use overapproximations that are too coarse.
Revised: Our evaluation demonstrates that ReﬁneAI can certify robustness properties beyond the reach of existing state-of-the-art complete veriﬁers (these can fail due to scalability issues) while simultaneously improving on the precision of existing incomplete veriﬁers (which can fail due to using too coarse of an overapproximation).

Original: We believe combining the strengths of overapproximation methods with that of mixed integer linear programming as done in this work is a promising direction for further advancing the state-of-the-art in neural network certiﬁcation.
Revised: Overall, we believe combining the strengths of overapproximation methods with those of mixed integer linear programming as done in this work is a promising direction for further advancing the state-of-the-art in neural network veriﬁcation.