Original: C ERTIFIED D EFENSES AGAINST A DVERSARIAL E X - AMPLES
Revised: C ERTIFIED D EFENSES AGAINST A DVERSARIAL E XAMPLES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While neural networks have achieved high accuracy on standard image classiﬁcation benchmarks, their accuracy drops to nearly zero in the presence of small adversarial perturbations to test inputs.
Revised: While neural networks have achieved high accuracy on standard image classiﬁcation benchmarks, their accuracy drops to nearly zero in the presence of small adversarial perturbations to test inputs.

Original: Defenses based on regularization and adversarial training have been proposed, but often followed by new, stronger attacks that defeat these defenses.
Revised: Defenses based on regularization and adversarial training have been proposed, but often followed by new, stronger attacks that defeat these defenses.

Original: Can we somehow end this arms race?
Revised: Can we somehow end this arms race?

Original: In this work, we study this problem for neural networks with one hidden layer.
Revised: In this work, we study this problem for neural networks with one hidden layer.

Original: We ﬁrst propose a method based on a semideﬁnite relaxation that outputs a certiﬁcate that for a given network and test input, no attack can force the error to exceed a certain value.
Revised: We ﬁrst propose a method based on a semideﬁnite relaxation that outputs a certiﬁcate that for a given network and test input, no attack can force the error to exceed a certain value.

Original: Second, as this certiﬁcate is differentiable, we jointly optimize it with the network parameters, providing an adaptive regularizer that encourages robustness against all attacks.
Revised: Second, as this certiﬁcate is differentiable, we jointly optimize it with the network parameters, providing an adaptive regularizer that encourages robustness against all attacks.

Original: On MNIST, our approach produces a network and a certiﬁcate that no attack that perturbs each pixel by at most (cid:15)
Revised: On MNIST, our approach produces a network and a certiﬁcate that no attack that perturbs each pixel by at most (cid:15)

Original: = 0 .
Revised: = 0 .

Original: 1 can cause more than 35% test error.
Revised: 1 can cause more than 35% test error.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Despite the impressive (and sometimes even superhuman) accuracies of machine learning on diverse tasks such as object recognition (He et al., 2015), speech recognition (Xiong et al., 2016), and playing Go (Silver et al., 2016), classiﬁers still fail catastrophically in the presence of small imperceptible but adversarial perturbations (Szegedy et al., 2014; Goodfellow et al., 2015; Kurakin et al., 2016).
Revised: Despite the impressive (and sometimes even superhuman) accuracies of machine learning on diverse tasks such as object recognition (He et al., 2015), speech recognition (Xiong et al., 2016), and playing Go (Silver et al., 2016), classiﬁers still fail catastrophically in the presence of small imperceptible but adversarial perturbations (Szegedy et al., 2014; Goodfellow et al., 2015; Kurakin et al., 2016).

Original: In addition to being an intriguing phenonemon, the existence of such “adversarial examples” exposes a serious vulnerability in current ML systems (Evtimov et al., 2017;
Revised: In addition to being an intriguing phenonemon, the existence of such “adversarial examples” exposes a serious vulnerability in current ML systems (Evtimov et al., 2017;

Original: Sharif et al., 2016; Carlini et al., 2016).
Revised: Sharif et al., 2016; Carlini et al., 2016).

Original: While formally deﬁning an “imperceptible” perturbation is difﬁcult, a commonly-used proxy is perturbations that are bounded in (cid:96) ∞
Revised: While formally deﬁning an “imperceptible” perturbation is difﬁcult, a commonly-used proxy is perturbations that are bounded in (cid:96) ∞

Original: -norm (Goodfellow et al., 2015; Madry et al., 2017; Tramèr et al., 2017); we focus on this attack model in this paper, as even for this proxy it is not known how to construct high-performing image classiﬁers that are robust to perturbations.
Revised: -norm (Goodfellow et al., 2015; Madry et al., 2017; Tramèr et al., 2017); we focus on this attack model in this paper, as even for this proxy it is not known how to construct high-performing image classiﬁers that are robust to perturbations.

Original: While a proposed defense (classiﬁer) is often empirically shown to be successful against the set of attacks known at the time, new stronger attacks are subsequently discovered that render the defense useless.
Revised: While a proposed defense (classiﬁer) is often empirically shown to be successful against the set of attacks known at the time, new stronger attacks are subsequently discovered that render the defense useless.

Original: For example, defensive distillation (Papernot et al., 2016c) and adversarial training against the Fast Gradient Sign Method (Goodfellow et al., 2015) were two defenses that were later shown to be ineffective against stronger attacks (Carlini & Wagner, 2016;
Revised: For example, defensive distillation (Papernot et al., 2016c) and adversarial training against the Fast Gradient Sign Method (Goodfellow et al., 2015) were two defenses that were later shown to be ineffective against stronger attacks (Carlini & Wagner, 2016;

Original: Tramèr et al., 2017).
Revised: Tramèr et al., 2017).

Original: In order to break this arms race between attackers and defenders, we need to come up with defenses that are successful against all attacks within a certain class.
Revised: In order to break this arms race between attackers and defenders, we need to come up with defenses that are successful against all attacks within a certain class.

Original: However, even computing the worst-case error for a given network against all adversarial perturbations in an (cid:96) ∞ -ball is computationally intractable.
Revised: However, even computing the worst-case error for a given network against all adversarial perturbations in an (cid:96) ∞ -ball is computationally intractable.

Original: One common approximation is to replace the worst-case loss with the loss from a given heuristic attack strategy, such as the Fast Gradient Sign
Revised: One common approximation is to replace the worst-case loss with the loss from a given heuristic attack strategy, such as the Fast Gradient Sign

Original: Method (Goodfellow et al., 2015) or more powerful iterative methods (Carlini & Wagner, 2017a;
Revised: Method (Goodfellow et al., 2015) or more powerful iterative methods (Carlini & Wagner, 2017a;

Original: Madry et al., 2017).
Revised: Madry et al., 2017).

Original: Adversarial training minimizes the loss with respect to these heuristics.
Revised: Adversarial training minimizes the loss with respect to these heuristics.

Original: However, this essentially minimizes a lower bound on the worst-case loss, which is problematic since points where the bound is loose have disproportionately lower objective values, which could lure and mislead an optimizer.
Revised: However, this essentially minimizes a lower bound on the worst-case loss, which is problematic since points where the bound is loose have disproportionately lower objective values, which could lure and mislead an optimizer.

Original: Indeed, while adversarial training often provides robustness against a speciﬁc attack, it often fails to generalize to new attacks, as described above.
Revised: Indeed, while adversarial training often provides robustness against a speciﬁc attack, it often fails to generalize to new attacks, as described above.

Original: Another approach is to compute the worst-case perturbation exactly using discrete optimization (Katz et al., 2017a;Carlini et al., 2017).
Revised: Another approach is to compute the worst-case perturbation exactly using discrete optimization (Katz et al., 2017a;

Original: Currently, these approaches can take up to several hours or longer to compute the loss
Revised: Carlini

Original: [Figure]
Revised: [Figure]

Original:  for a single example even for small networks with a few hundred hidden units.
Revised: et al., 2017). Currently, these approaches can take up to several hours or longer to compute the loss for a single example even for small networks with a few hundred hidden units.

Original: Training a network would require performing this computation in the inner loop, which is infeasible.
Revised: Training a network would require performing this computation in the inner loop, which is infeasible.

Original: In this paper, we introduce an approach that avoids both the inaccuracy of lower bounds and the intractability of exact computation, by computing an upper bound on the worst-case loss for neural networks with one hidden layer, based on a semideﬁnite relaxation that can be computed efﬁciently.
Revised: In this paper, we introduce an approach that avoids both the inaccuracy of lower bounds and the intractability of exact computation, by computing an upper bound on the worst-case loss for neural networks with one hidden layer, based on a semideﬁnite relaxation that can be computed efﬁciently.

Original: This upper bound serves as a certiﬁcate of robustness against all attacks for a given network and input.
Revised: This upper bound serves as a certiﬁcate of robustness against all attacks for a given network and input.

Original: Minimizing an upper bound is safer than minimizing a lower bound, because points where the bound is loose have disproportionately higher objective values, which the optimizer will tend to avoid.
Revised: Minimizing an upper bound is safer than minimizing a lower bound, because points where the bound is loose have disproportionately higher objective values, which the optimizer will tend to avoid.

Original: Furthermore, our certiﬁcate of robustness, by virtue of being differentiable, is trainable —it can be optimized at training time jointly with the network, acting as a regularizer that encourages robustness against all (cid:96) ∞ attacks.
Revised: Furthermore, our certiﬁcate of robustness, by virtue of being differentiable, is trainable —it can be optimized at training time jointly with the network, acting as a regularizer that encourages robustness against all (cid:96) ∞ attacks.

Original: In summary, we are the ﬁrst  to demonstrate a certiﬁable, trainable, and scalable method for defending against adversarial examples on two-layer networks.
Revised: In summary, we are the ﬁrst (along with the concurrent work of Kolter & Wong (2017)) to demonstrate a certiﬁable, trainable, and scalable method for defending against adversarial examples on two-layer networks.

Original: We train a network on MNIST whose test error on clean data is 4 . 2% , and which comes with a certiﬁcate that no attack can misclassify more than 35% of the test examples using (cid:96) ∞ perturbations of size (cid:15)
Revised: We train a network on MNIST whose test error on clean data is 4 . 2% , and which comes with a certiﬁcate that no attack can misclassify more than 35% of the test examples using (cid:96) ∞ perturbations of size (cid:15)

Original: = 0 .
Revised: = 0 .

Original: 1 ; for comparison, the only other certiﬁed error bound we are aware of is 60% at (cid:15) = 0 .

Original: 05 , and no network (certiﬁed or not) has error better than 11% at (cid:15)

Original: = 0 .

Original: Notation.
Revised: Notation.

Original: For a vector z ∈ R n , we use z i to denote the i th coordinate of z .
Revised: For a vector z ∈ R n , we use z i to denote the i th coordinate of z .

Original: For a matrix Z ∈ R m × n ,
Revised: For a matrix Z ∈ R m × n ,

Original: Z i denotes the i th row.
Revised: Z i denotes the i th row.

Original: For any activation function σ : R → R (e.g., sigmoid, ReLU) and a vector z ∈ R n , σ ( z ) is a vector in R n with σ ( z ) i =
Revised: For any activation function σ : R → R (e.g., sigmoid, ReLU) and a vector z ∈ R n , σ ( z ) is a vector in R n with σ ( z ) i =

Original: σ ( z i ) (non-linearity is applied element-wise).
Revised: σ ( z i ) (non-linearity is applied element-wise).

Original: We use B (cid:15) ( z ) to denote the (cid:96) ∞ ball of radius (cid:15) around z ∈ R d : B (cid:15) ( z ) =
Revised: We use B (cid:15) ( z ) to denote the (cid:96) ∞ ball of radius (cid:15) around z ∈ R d : B (cid:15) ( z ) =

Original: { ˜ z | | ˜ z i − z i | ≤ (cid:15) for i = 1 , 2 , . . .
Revised: { ˜ z | | ˜ z i − z i | ≤ (cid:15) for i = 1 , 2 , . . .

Original: d } .
Revised: d } .

Original: Finally, we denote the vector of all zeros by 0 and the vector of all ones by 1 .
Revised: Finally, we denote the vector of all zeros by 0 and the vector of all ones by 1 .

Original:  S ETUP
Revised:  S ETUP

Original: Score-based classiﬁers.
Revised: Score-based classiﬁers.

Original: Our goal is to learn a mapping C : X → Y , where X = R d is the input space (e.g., images) and Y = { 1 , . . .
Revised: Our goal is to learn a mapping C : X → Y , where X = R d is the input space (e.g., images) and Y = { 1 , . . .

Original: , k } is the set of k class labels (e.g., object categories).
Revised: , k } is the set of k class labels (e.g., object categories).

Original: Assume
Revised: Assume

Original: C is driven by a scoring function f
Revised: C is driven by a scoring function f

Original: i : X → R for all classes i ∈ Y , where the classiﬁer chooses the class with the highest score: C ( x ) = arg max i ∈Y f i ( x ) .
Revised: i : X → R for all classes i ∈ Y , where the classiﬁer chooses the class with the highest score: C ( x ) = arg max i ∈Y f i ( x ) .

Original: Also, deﬁne the pairwise margin f ij ( x ) def = f i ( x ) − f j ( x ) for every pair of classes ( i, j ) .
Revised: Also, deﬁne the pairwise margin f ij ( x ) def = f i ( x ) − f j ( x ) for every pair of classes ( i, j ) .

Original: Note that the classiﬁer outputs C ( x ) =
Revised: Note that the classiﬁer outputs C ( x ) =

Original: i iff f ij ( x ) > 0 for all alternative classes j (cid:54)
Revised: i iff f ij ( x ) > 0 for all alternative classes j (cid:54)

Original: =
Revised: =

Original: i .
Revised: i .

Original: Normally, a classiﬁer is evaluated on the 0-1 loss (cid:96) ( x, y )
Revised: Normally, a classiﬁer is evaluated on the 0-1 loss (cid:96) ( x, y )

Original: =
Revised: =

Original: I [ C ( x ) (cid:54) =
Revised: I [ C ( x ) (cid:54) =

Original: y ] .
Revised: y ] .

Original: This paper focuses on linear classiﬁers and neural networks with one hidden layer.
Revised: This paper focuses on linear classiﬁers and neural networks with one hidden layer.

Original: For linear classiﬁers, f i ( x ) def = W (cid:62) i x , where W i is the i th row of the parameter matrix W ∈ R k × d .
Revised: For linear classiﬁers, f i ( x ) def = W (cid:62) i x , where W i is the i th row of the parameter matrix W ∈ R k × d .

Original: For neural networks with one hidden layer consisting of m hidden units, the scoring function is f i ( x ) =
Revised: For neural networks with one hidden layer consisting of m hidden units, the scoring function is f i ( x ) =

Original: V (cid:62) i σ ( Wx ) , where W ∈ R m × d and V ∈ R k × m are the parameters of the ﬁrst and second layer, respectively, and σ is a non-linear activation function applied elementwise (e.g., for ReLUs, σ ( z )
Revised: V (cid:62) i σ ( Wx ) , where W ∈ R m × d and V ∈ R k × m are the parameters of the ﬁrst and second layer, respectively, and σ is a non-linear activation function applied elementwise (e.g., for ReLUs, σ ( z )

Original: = max( z, 0) ).
Revised: = max( z, 0) ).

Original: We will assume below that the gradients of σ are bounded: σ (cid:48)
Revised: We will assume below that the gradients of σ are bounded: σ (cid:48)

Original: ( z ) ∈ [0 , 1] for all z ∈ R ; this is true for ReLUs, as well as for sigmoids (with the stronger bound σ (cid:48)
Revised: ( z ) ∈ [0 , 1] for all z ∈ R ; this is true for ReLUs, as well as for sigmoids (with the stronger bound σ (cid:48)

Original: ( z ) ∈ [0 , 14 ] ).
Revised: ( z ) ∈ [0 , 14 ] ).

Original: Attack model.
Revised: Attack model.

Original: We are interested in classiﬁcation in the presence of an attacker A : X → X that takes a (test) input x and returns a perturbation ˜ x .
Revised: We are interested in classiﬁcation in the presence of an attacker A : X → X that takes a (test) input x and returns a perturbation ˜ x .

Original: We consider attackers A that can perturb each feature x i by at most (cid:15) ≥ 0 ; formally, A ( x ) is required to lie in the (cid:96) ∞ ball B (cid:15) ( x ) def = { ˜ x | (cid:107) ˜ x − x (cid:107) ∞ ≤ (cid:15) } , which is the standard constraint ﬁrst proposed in Szegedy et al.
Revised: We consider attackers A that can perturb each feature x i by at most (cid:15) ≥ 0 ; formally, A ( x ) is required to lie in the (cid:96) ∞ ball B (cid:15) ( x ) def = { ˜ x | (cid:107) ˜ x − x (cid:107) ∞ ≤ (cid:15) } , which is the standard constraint ﬁrst proposed in Szegedy et al.

Original: Deﬁne the adversarial loss with respect to A as (cid:96) A
Revised: Deﬁne the adversarial loss with respect to A as (cid:96) A

Original: ( x, y )
Revised: ( x, y )

Original: =
Revised: =

Original: I [ C ( A ( x )) (cid:54) =
Revised: I [ C ( A ( x )) (cid:54) =

Original: y ] .
Revised: y ] .

Original: We assume the white-box setting, where the attacker A has full knowledge of C .
Revised: We assume the white-box setting, where the attacker A has full knowledge of C .

Original: The optimal (untargeted) attack chooses the input that maximizes the pairwise margin of an incorrect class i over the correct class y :
Revised: The optimal (untargeted) attack chooses the input that maximizes the pairwise margin of an incorrect class i over the correct class y :

Original: A opt ( x ) = arg max ˜ x ∈ B (cid:15) ( x ) max i f iy (˜ x ) .
Revised: A opt ( x ) = arg max ˜ x ∈ B (cid:15) ( x ) max i f iy (˜ x ) .

Original: For a neural network, computing A opt is a non-convex optimization problem; heuristics are typically employed, such as the Fast Gradient
Revised: For a neural network, computing A opt is a non-convex optimization problem; heuristics are typically employed, such as the Fast Gradient

Original: Sign Method (FGSM) (Goodfellow et al., 2015), which perturbs x based on the gradient, or the
Revised: Sign Method (FGSM) (Goodfellow et al., 2015), which perturbs x based on the gradient, or the

Original: Carlini-Wagner attack, which performs iterative optimization (Carlini & Wagner, 2017b).
Revised: Carlini-Wagner attack, which performs iterative optimization (Carlini & Wagner, 2017b).

Original:  C ERTIFICATE ON THE ADVERSARIAL LOSS
Revised:  C ERTIFICATE ON THE ADVERSARIAL LOSS

Original: For ease of exposition, we ﬁrst consider binary classiﬁcation with classes Y = { 1 , 2 } ; the multiclass extension is discussed at the end of Section 3.3.
Revised: For ease of exposition, we ﬁrst consider binary classiﬁcation with classes Y = { 1 , 2 } ; the multiclass extension is discussed at the end of Section 3.3.

Original: Without loss of generality, assume the correct label for x is y = 2 .
Revised: Without loss of generality, assume the correct label for x is y = 2 .

Original: Simplifying notation, let f ( x ) =
Revised: Simplifying notation, let f ( x ) =

Original: f 1 ( x ) − f 2 ( x ) be the margin of the incorrect class over the correct class.
Revised: f 1 ( x ) − f 2 ( x ) be the margin of the incorrect class over the correct class.

Original: Then A opt ( x ) = arg max ˜ x ∈ B (cid:15) ( x ) f (˜ x ) is the optimal attack, which is successful if f ( A opt ( x )) > 0 .
Revised: Then A opt ( x ) = arg max ˜ x ∈ B (cid:15) ( x ) f (˜ x ) is the optimal attack, which is successful if f ( A opt ( x )) > 0 .

Original: Since f ( A opt ( x )) is intractable to compute, we will try to upper bound it via a tractable relaxation.
Revised: Since f ( A opt ( x )) is intractable to compute, we will try to upper bound it via a tractable relaxation.

Original: In the rest of this section, we ﬁrst review a classic result in the simple case of linear networks where a tight upper bound is based on the (cid:96) 1 -norm of the weights (Section 3.1).
Revised: In the rest of this section, we ﬁrst review a classic result in the simple case of linear networks where a tight upper bound is based on the (cid:96) 1 -norm of the weights (Section 3.1).

Original: We then extend this to general classiﬁers, in which f
Revised: We then extend this to general classiﬁers, in which f

Original: ( A opt ( x )) can be upper bounded using the maximum (cid:96) 1 -norm of the gradient at any point ˜ x ∈ B (cid:15) ( x ) (Section 3.2).
Revised: ( A opt ( x )) can be upper bounded using the maximum (cid:96) 1 -norm of the gradient at any point ˜ x ∈ B (cid:15) ( x ) (Section 3.2).

Original: For two-layer networks, this quantity is upper bounded by the optimal value f QP ( x ) of a non-convex quadratic program (QP)
Revised: For two-layer networks, this quantity is upper bounded by the optimal value f QP ( x ) of a non-convex quadratic program (QP)

Original: (Section 3.3), which in turn is upper bounded by the optimal value f SDP ( x ) of a semideﬁnite program (SDP).
Revised: (Section 3.3), which in turn is upper bounded by the optimal value f SDP ( x ) of a semideﬁnite program (SDP).

Original: The SDP is convex and can be computed exactly (which is important for obtainining actual certiﬁcates).
Revised: The SDP is convex and can be computed exactly (which is important for obtainining actual certiﬁcates).

Original: To summarize, we have the following chain of inequalities:
Revised: To summarize, we have the following chain of inequalities:

Original: [Equation]
Revised: [Equation]

Original: which implies that the adversarial loss (cid:96) A ( x )
Revised: which implies that the adversarial loss (cid:96) A ( x )

Original: =
Revised: =

Original: I [ f ( A ( x )) > 0] with respect to any attacker A is upper bounded by I [ f SDP ( x ) > 0] .
Revised: I [ f ( A ( x )) > 0] with respect to any attacker A is upper bounded by I [ f SDP ( x ) > 0] .

Original: Note that for certain non-linearities such as ReLUs, ∇ f
Revised: Note that for certain non-linearities such as ReLUs, ∇ f

Original: (˜ x ) does not exist everywhere, but our analysis below holds as long as f is differentiable almost-everywhere.
Revised: (˜ x ) does not exist everywhere, but our analysis below holds as long as f is differentiable almost-everywhere.

Original: 3. L INEAR CLASSIFIERS
Revised: L INEAR CLASSIFIERS

Original: For (binary) linear classiﬁers, we have f ( x ) =
Revised: For (binary) linear classiﬁers, we have f ( x ) =

Original: ( W 1 − W 2 ) (cid:62) x , where
Revised: ( W 1 − W 2 ) (cid:62) x , where

Original: W 1 , W 2 ∈ R d are the weight vectors for the two classes.
Revised: W 1 , W 2 ∈ R d are the weight vectors for the two classes.

Original: For any input ˜ x ∈ B (cid:15) ( x ) , Hölder’s inequality with (cid:107) x − ˜ x (cid:107) ∞ ≤ (cid:15) gives:
Revised: For any input ˜ x ∈ B (cid:15) ( x ) , Hölder’s inequality with (cid:107) x − ˜ x (cid:107) ∞ ≤ (cid:15) gives:

Original: [Equation]
Revised: [Equation]

Original: Note that this bound is tight, obtained by taking A opt ( x ) i
Revised: Note that this bound is tight, obtained by taking A opt ( x ) i

Original: = x i + (cid:15) sign( W 1 i − W 2 i ) .
Revised: = x i + (cid:15) sign( W 1 i − W 2 i ) .

Original: 3. G ENERAL CLASSIFIERS
Revised: 3. G ENERAL CLASSIFIERS

Original: For more general classiﬁers, we cannot compute f ( A opt ( x )) exactly, but motivated by the above, we can use the gradient to obtain a linear approximation g :[Equation]
Revised: For more general classiﬁers, we cannot compute f ( A opt ( x )) exactly, but motivated by the above, we can use the gradient to obtain a linear approximation g :

Original: [Equation]
Revised: [Equation]

Original: Using this linear approximation to generate A ( x ) corresponds exactly to the Fast Gradient Sign
Revised: Using this linear approximation to generate A ( x ) corresponds exactly to the Fast Gradient Sign

Original: Method (FGSM) (Goodfellow et al., 2015).
Revised: Method (FGSM) (Goodfellow et al., 2015).

Original: However, f is only close to g when ˜ x is very close to x , and people have observed the gradient masking phenomenon (Tramèr et al., 2017; Papernot et al., 2016b) in several proposed defenses that train against approximations like g , such as saturating networks (Nayebi & Ganguli, 2017), distillation (Papernot et al., 2016c), and adversarial training (Goodfellow et al., 2015).
Revised: However, f is only close to g when ˜ x is very close to x , and people have observed the gradient masking phenomenon (Tramèr et al., 2017; Papernot et al., 2016b) in several proposed defenses that train against approximations like g , such as saturating networks (Nayebi & Ganguli, 2017), distillation (Papernot et al., 2016c), and adversarial training (Goodfellow et al., 2015).

Original: Speciﬁcally, defenses that try to minimize (cid:107)∇ f ( x ) (cid:107) 1 locally at the training points result in loss surfaces that exhibit sharp curvature near those points, essentially rendering the linear approximation g (˜ x ) meaningless.
Revised: Speciﬁcally, defenses that try to minimize (cid:107)∇ f ( x ) (cid:107) 1 locally at the training points result in loss surfaces that exhibit sharp curvature near those points, essentially rendering the linear approximation g (˜ x ) meaningless.

Original: Some attacks (Carlini & Wagner, 2016; Tramèr et al., 2017) evade these defenses and witness a large f ( A opt ( x )) .
Revised: Some attacks (Carlini & Wagner, 2016; Tramèr et al., 2017) evade these defenses and witness a large f ( A opt ( x )) .

Original: Figure 1a provides a simple illustration.
Revised: Figure 1a provides a simple illustration.

Original: We propose an alternative approach: use integration to obtain an exact expression for f (˜ x ) in terms of the gradients along the line between x and ˜ x : 
Revised: We propose an alternative approach: use integration to obtain an exact expression for f (˜ x ) in terms of the gradients along the line between x and ˜ x : [Equation]

Original: [Equation]
Revised: [Equation]

Original: (4) where the inequality follows from the fact that t ˜ x + (1 − t ) x ∈ B (cid:15) ( x ) for all t ∈ [0 , 1] .
Revised: where the inequality follows from the fact that t ˜ x + (1 − t ) x ∈ B (cid:15) ( x ) for all t ∈ [0 , 1] .

Original: The key difference between (4) and (3) is that we consider the gradients over the entire ball B (cid:15) ( x ) rather than only at x (Figure 1b).
Revised: The key difference between (4) and (3) is that we consider the gradients over the entire ball B (cid:15) ( x ) rather than only at x (Figure 1b).

Original: However, computing the RHS of (4) is intractable in general.
Revised: However, computing the RHS of (4) is intractable in general.

Original: For two-layer neural networks, this optimization has additional structure which we will exploit in the next section.
Revised: For two-layer neural networks, this optimization has additional structure which we will exploit in the next section.

Original: 3. T WO - LAYER NEURAL NETWORKS
Revised: 3. T WO - LAYER NEURAL NETWORKS

Original: We now unpack the upper bound (4) for two-layer neural networks.
Revised: We now unpack the upper bound (4) for two-layer neural networks.

Original: Recall from Section 2 that f ( x ) =
Revised: Recall from Section 2 that f ( x ) =

Original: f 1 ( x ) − f 2 ( x ) = v (cid:62) σ ( Wx ) , where v def = V 1 − V 2 ∈ R m is the difference in second-layer weights for the two classes.
Revised: f 1 ( x ) − f 2 ( x ) = v (cid:62) σ ( Wx ) , where v def = V 1 − V 2 ∈ R m is the difference in second-layer weights for the two classes.

Original: Let us try to bound the norm of the gradient (cid:107)∇ f (˜ x ) (cid:107) 1 for ˜ x ∈ B (cid:15) ( x ) .
Revised: Let us try to bound the norm of the gradient (cid:107)∇ f (˜ x ) (cid:107) 1 for ˜ x ∈ B (cid:15) ( x ) .

Original: If we apply the chain rule, we see that the only dependence on ˜ x is σ (cid:48)
Revised: If we apply the chain rule, we see that the only dependence on ˜ x is σ (cid:48)

Original: ( W ˜ x ) , the activation derivatives.
Revised: ( W ˜ x ) , the activation derivatives.

Original: We now leverage our assumption that σ (cid:48) ( z ) ∈ [0 , 1] m for all vectors z ∈ R m , so that we can optimize over possible activation derivatives s ∈ [0 , 1] m directly independent of x (note that there is potential looseness because not all such s need be obtainable via some ˜ x ∈ B (cid:15) ( x ) ).
Revised: We now leverage our assumption that σ (cid:48) ( z ) ∈ [0 , 1] m for all vectors z ∈ R m , so that we can optimize over possible activation derivatives s ∈ [0 , 1] m directly independent of x (note that there is potential looseness because not all such s need be obtainable via some ˜ x ∈ B (cid:15) ( x ) ).

Original: Therefore:
Revised: Therefore:

Original: [Equation]
Revised: [Equation]

Original: where (i) follows from the chain rule, (ii) uses the fact that σ has bounded derivatives σ (cid:48)
Revised: where (i) follows from the chain rule, (ii) uses the fact that σ has bounded derivatives σ (cid:48)

Original: ( z ) ∈ [0 , 1] , and (iii) follows from the identity (cid:107) z (cid:107) 1 = max t ∈ [ − 1 , 1] d t (cid:62) z .
Revised: ( z ) ∈ [0 , 1] , and (iii) follows from the identity (cid:107) z (cid:107) 1 = max t ∈ [ − 1 , 1] d t (cid:62) z .

Original: (Note that for sigmoid networks, where σ (cid:48)
Revised: (Note that for sigmoid networks, where σ (cid:48)

Original: ( z ) ∈ [0 , 14 ] , we can strengthen the above bound by a corresponding factor of 1 4 .)
Revised: ( z ) ∈ [0 , 14 ] , we can strengthen the above bound by a corresponding factor of 1 4 .)

Original: Substituting the bound (5) into (4), we obtain an upper bound on the adversarial loss that we call f QP :
Revised: Substituting the bound (5) into (4), we obtain an upper bound on the adversarial loss that we call f QP :

Original: [Equation]
Revised: [Equation]

Original: Unfortunately, (6) still involves a non-convex optimization problem (since W (cid:62) diag( v ) is not necessarily negative semideﬁnite).
Revised: Unfortunately, (6) still involves a non-convex optimization problem (since W (cid:62) diag( v ) is not necessarily negative semideﬁnite).

Original: In fact, it is similar to the NP-hard MAXCUT problem, which requires maximizing x (cid:62) Lx over x ∈ [ − 1 , 1] d for a graph with Laplacian matrix L .
Revised: In fact, it is similar to the NP-hard MAXCUT problem, which requires maximizing x (cid:62) Lx over x ∈ [ − 1 , 1] d for a graph with Laplacian matrix L .

Original: While MAXCUT is NP-hard, it can be efﬁciently approximated, as shown by the celebrated semidefinite programming relaxation for MAXCUT in Goemans & Williamson (1995).
Revised: While MAXCUT is NP-hard, it can be efﬁciently approximated, as shown by the celebrated semidefinite programming relaxation for MAXCUT in Goemans & Williamson (1995).

Original: We follow a similar approach here to obtain an upper bound on f QP ( x ) .
Revised: We follow a similar approach here to obtain an upper bound on f QP ( x ) .

Original: First, to make our variables lie in [ − 1 , 1] m instead of [0 , 1] m , we reparametrize s to produce:
Revised: First, to make our variables lie in [ − 1 , 1] m instead of [0 , 1] m , we reparametrize s to produce:

Original: [Equation]
Revised: [Equation]

Original: Next pack the variables into a vector y ∈ R m + d +1 and the parameters into a matrix M :
Revised: Next pack the variables into a vector y ∈ R m + d +1 and the parameters into a matrix M :

Original: [Equation]
Revised: [Equation]

Original: In terms of these new objects, our objective takes the form:
Revised: In terms of these new objects, our objective takes the form:

Original: [Equation]
Revised: [Equation]

Original: Note that every valid vector y ∈ [ − 1 , +1] m + d [Equation] 1 satisﬁes the constraints yy (cid:62) (cid:23) 

Revised: Note that every valid vector y ∈ [ − 1 , +1] m + d +1 satisﬁes the constraints yy (cid:62) (cid:23)

Original: and ( yy (cid:62) )
Revised: and ( yy (cid:62) )

Original: jj = 1 .
Revised: jj = 1 .

Original: Deﬁning P =
Revised: Deﬁning P =

Original: yy (cid:62) , we obtain the following convex semideﬁnite relaxation of our problem:
Revised: yy (cid:62) , we obtain the following convex semideﬁnite relaxation of our problem:

Original: [Equation]
Revised: [Equation]

Original: Note that the optimization of the semideﬁnite program depends only on the weights v and W and does not depend on the inputs x , so it only needs to be computed once for a model ( v, W ) .
Revised: Note that the optimization of the semideﬁnite program depends only on the weights v and W and does not depend on the inputs x , so it only needs to be computed once for a model ( v, W ) .

Original: Semideﬁnite programs can be solved with off-the-shelf optimizers, although these optimizers are somewhat slow on large instances.
Revised: Semideﬁnite programs can be solved with off-the-shelf optimizers, although these optimizers are somewhat slow on large instances.

Original: In Section 4 we propose a fast stochastic method for training, which only requires computing the top eigenvalue of a matrix.
Revised: In Section 4 we propose a fast stochastic method for training, which only requires computing the top eigenvalue of a matrix.

Original: Generalization to multiple classes.
Revised: Generalization to multiple classes.

Original: The preceding arguments all generalize to the pairwise margins f ij , to give:
Revised: The preceding arguments all generalize to the pairwise margins f ij , to give:

Original: [Equation]
Revised: [Equation]

Original: M ij ( V, W ) is deﬁned as in (9) with v =
Revised: M ij ( V, W ) is deﬁned as in (9) with v =

Original: V i − V j .
Revised: V i − V j .

Original: The adversarial loss of any attacker, (cid:96) A ( x, y )
Revised: The adversarial loss of any attacker, (cid:96) A ( x, y )

Original: =
Revised: =

Original: I [max i (cid:54) =
Revised: I [max i (cid:54) =

Original: y f iy ( A ( x )) > 0] , can be bounded using the fact that f iy SDP ( x ) ≥ f iy ( A ( x )) .
Revised: y f iy ( A ( x )) > 0] , can be bounded using the fact that f iy SDP ( x ) ≥ f iy ( A ( x )) .

Original: In particular,
Revised: In particular,

Original: [Equation]
Revised: [Equation]

Original:  T RAINING THE CERTIFICATE
Revised:  T RAINING THE CERTIFICATE

Original: In the previous section, we proposed an upper bound (12) on the loss (cid:96)
Revised: In the previous section, we proposed an upper bound (12) on the loss (cid:96)

Original: A ( x, y ) of any attack A, based on the bound (11).
Revised: A ( x, y ) of any attack A, based on the bound (11).

Original: Normal training with some classiﬁcation loss (cid:96) cls ( V, W ; x n , y n ) like hinge loss or cross-entropy will encourage the pairwise margin f ij ( x ) to be large in magnitude, but won’t necessarily cause the second term in (11) involving M ij to be small.
Revised: Normal training with some classiﬁcation loss (cid:96) cls ( V, W ; x n , y n ) like hinge loss or cross-entropy will encourage the pairwise margin f ij ( x ) to be large in magnitude, but won’t necessarily cause the second term in (11) involving M ij to be small.

Original: A natural strategy is thus to use the following regularized objective given training examples ( x n , y n ) , which pushes down on both terms:
Revised: A natural strategy is thus to use the following regularized objective given training examples ( x n , y n ) , which pushes down on both terms:

Original: [Equation]
Revised: [Equation]

Original: where λ ij > 0 are the regularization hyperparameters.
Revised: where λ ij > 0 are the regularization hyperparameters.

Original: However, computing the gradients of the above objective involves ﬁnding the optimal solution of a semideﬁnite program, which is slow.
Revised: However, computing the gradients of the above objective involves ﬁnding the optimal solution of a semideﬁnite program, which is slow.

Original: Duality to the rescue.
Revised: Duality to the rescue.

Original: Our computational burden is lifted by the beautiful theory of duality, which provides the following equivalence between the primal maximization problem over P , and a dual minimization problem over new variables
Revised: Our computational burden is lifted by the beautiful theory of duality, which provides the following equivalence between the primal maximization problem over P , and a dual minimization problem over new variables

Original: c (see Section A for details):
Revised: c (see Section A for details):

Original: [Equation]
Revised: [Equation]

Original: where D =
Revised: where D =

Original: ( d + m + 1) and λ +max ( B ) is the maximum eigenvalue of B , or 0 if all eigenvalues are negative.
Revised: ( d + m + 1) and λ +max ( B ) is the maximum eigenvalue of B , or 0 if all eigenvalues are negative.

Original: This dual formulation allows us to introduce additional dual variables c ij ∈ R D that are optimized at the same time as the parameters V and W , resulting in an objective that can be trained efﬁciently using stochastic gradient methods.
Revised: This dual formulation allows us to introduce additional dual variables c ij ∈ R D that are optimized at the same time as the parameters V and W , resulting in an objective that can be trained efﬁciently using stochastic gradient methods.

Original: The ﬁnal objective.
Revised: The ﬁnal objective.

Original: Using (14), we end up optimizing the following training objective:
Revised: Using (14), we end up optimizing the following training objective:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The objective in (15) can be optimized efﬁciently.
Revised: The objective in (15) can be optimized efﬁciently.

Original: The most expensive operation is λ +max , which requires computing the maximum eigenvector of the matrix M ij − diag( c ij ) in order to take gradients.
Revised: The most expensive operation is λ +max , which requires computing the maximum eigenvector of the matrix M ij − diag( c ij ) in order to take gradients.

Original: This can be done efﬁciently using standard implementations of iterative methods like Lanczos.
Revised: This can be done efﬁciently using standard implementations of iterative methods like Lanczos.

Original: Further implementation details (including tuning of λ ij ) are presented in Section 6.3.
Revised: Further implementation details (including tuning of λ ij ) are presented in Section 6.3.

Original: Dual certiﬁcate of robustness.
Revised: Dual certiﬁcate of robustness.

Original: The dual formulation is also useful because any value of the dual is an upper bound on the optimal value of the primal.
Revised: The dual formulation is also useful because any value of the dual is an upper bound on the optimal value of the primal.

Original: Speciﬁcally, if ( W [ t ] , V [ t ] , c
Revised: Speciﬁcally, if ( W [ t ] , V [ t ] , c

Original: [ t ]) are the parameters at iteration t of training, then
Revised: [ t ]) are the parameters at iteration t of training, then

Original: [Equation]
Revised: [Equation]

Original: for any attack A .
Revised: for any attack A .

Original: As we train the network, we obtain a quick upper bound on the worst-case adversarial loss directly from the regularization loss , without having to optimize an SDP each time.
Revised: As we train the network, we obtain a quick upper bound on the worst-case adversarial loss directly from the regularization loss , without having to optimize an SDP each time.

Original:  O THER UPPER BOUNDS
Revised:  O THER UPPER BOUNDS

Original: In Section 3, we described a function f ij SDP that yields an efﬁcient upper bound on the adversarial loss, which we obtained using convex relaxations.
Revised: In Section 3, we described a function f ij SDP that yields an efﬁcient upper bound on the adversarial loss, which we obtained using convex relaxations.

Original: One could consider other simple ways to upper bound the loss; we describe here two common ones based on the spectral and Frobenius norms.
Revised: One could consider other simple ways to upper bound the loss; we describe here two common ones based on the spectral and Frobenius norms.

Original: Spectral bound: Note that v (cid:62) ( σ ( W ˜ x ) − σ ( Wx )) ≤ (cid:107) v (cid:107) 2 (cid:107) σ ( W ˜ x ) − σ ( Wx ) (cid:107) 2 by CauchySchwarz.
Revised: Spectral bound: Note that v (cid:62) ( σ ( W ˜ x ) − σ ( Wx )) ≤ (cid:107) v (cid:107) 2 (cid:107) σ ( W ˜ x ) − σ ( Wx ) (cid:107) 2 by CauchySchwarz.

Original: Moreover, since σ is contractive, (cid:107) σ ( W ˜ x ) − σ ( Wx ) (cid:107) 2 ≤ (cid:107) W
Revised: Moreover, since σ is contractive, (cid:107) σ ( W ˜ x ) − σ ( Wx ) (cid:107) 2 ≤ (cid:107) W

Original: [Equation]
Revised: [Equation]

Original: ˜

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: W
Revised: W

Original: [Equation]
Revised: [Equation]

Original: x (cid:107) 2 ≤ (cid:15) √ d (cid:107) W (cid:107) 2 , where (cid:107) W (cid:107) 2 is the spectral norm (maximum singular value) of W .
Revised: x (cid:107) 2 ≤ (cid:15) √ d (cid:107) W (cid:107) 2 , where (cid:107) W (cid:107) 2 is the spectral norm (maximum singular value) of W .

Original: This yields the following upper bound that we denote by f spectral :
Revised: This yields the following upper bound that we denote by f spectral :

Original: [Equation]
Revised: [Equation]

Original: This measure of vulnerability to adversarial examples based on the spectral norms of the weights of each layer is considered in Szegedy et al.
Revised: This measure of vulnerability to adversarial examples based on the spectral norms of the weights of each layer is considered in Szegedy et al.

Original: (2014) and Cisse et al.
Revised: (2014) and Cisse et al.

Original: Frobenius bound:
Revised: Frobenius bound:

Original: For ease in training, often the Frobenius norm is regularized (weight decay) instead of the spectral norm.
Revised: For ease in training, often the Frobenius norm is regularized (weight decay) instead of the spectral norm.

Original: Since (cid:107) W (cid:107) F ≥ (cid:107) W (cid:107) 2 , we get a corresponding upper bound f frobenius :
Revised: Since (cid:107) W (cid:107) F ≥ (cid:107) W (cid:107) 2 , we get a corresponding upper bound f frobenius :

Original: [Equation]
Revised: [Equation]

Original: In Section 6, we empirically compare our proposed bound using f ij SDP to these two upper bounds.
Revised: In Section 6, we empirically compare our proposed bound using f ij SDP to these two upper bounds.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluated our method on the MNIST dataset of handwritten digits, where the task is to classify images into one of ten classes.
Revised: We evaluated our method on the MNIST dataset of handwritten digits, where the task is to classify images into one of ten classes.

Original: Our results can be summarized as follows:
Revised: Our results can be summarized as follows:

Original: First, in Section 6.1, we show that our certiﬁcates of robustness are tighter than those based on simpler methods such as
Revised: First, in Section 6.1, we show that our certiﬁcates of robustness are tighter than those based on simpler methods such as

Original: Frobenius and spectral bounds (Section 5), but our bounds are still too high to be meaningful for general networks.
Revised: Frobenius and spectral bounds (Section 5), but our bounds are still too high to be meaningful for general networks.

Original: Then in Section 6.2, we show that by training on the certiﬁcates, we obtain networks with much better bounds and hence meaningful robustness.
Revised: Then in Section 6.2, we show that by training on the certiﬁcates, we obtain networks with much better bounds and hence meaningful robustness.

Original: This reﬂects an important point: while accurately analyzing the robustness of an arbitrary network is hard, training the certiﬁcate jointly leads to a network that is robust and certiﬁably so.
Revised: This reﬂects an important point: while accurately analyzing the robustness of an arbitrary network is hard, training the certiﬁcate jointly leads to a network that is robust and certiﬁably so.

Original: In Section 6.3, we present implementation details, design choices, and empirical observations that we made while implementing our method.
Revised: In Section 6.3, we present implementation details, design choices, and empirical observations that we made while implementing our method.

Original: Networks.
Revised: Networks.

Original: In this work, we focus on two layer networks.
Revised: In this work, we focus on two layer networks.

Original: In all our experiments, we used neural networks with m = 500 hidden units, and TensorFlow’s implementation of Adam (Kingma & Ba, 2014) as the optimizer; we considered networks with more hidden units, but these did not substantially improve accuracy.
Revised: In all our experiments, we used neural networks with m = 500 hidden units, and TensorFlow’s implementation of Adam (Kingma & Ba, 2014) as the optimizer; we considered networks with more hidden units, but these did not substantially improve accuracy.

Original: We experimented with both the multiclass hinge loss and cross-entropy.
Revised: We experimented with both the multiclass hinge loss and cross-entropy.

Original: All hyperparameters (including the choice of loss function) were tuned based on the error of the Projected Gradient Descent (PGD) attack (Madry et al., 2017) at (cid:15)
Revised: All hyperparameters (including the choice of loss function) were tuned based on the error of the Projected Gradient Descent (PGD) attack (Madry et al., 2017) at (cid:15)

Original: = 0 .
Revised: = 0 .

Original: 1 ; we report the hyperparameter settings below.
Revised: 1 ; we report the hyperparameter settings below.

Original: We considered the following training objectives providing 5 different networks:
Revised: We considered the following training objectives providing 5 different networks:

Original: Normal training (NT-NN).
Revised: Normal training (NT-NN).

Original: Cross-entropy loss and no explicit regularization.
Revised: Cross-entropy loss and no explicit regularization.

Original: Frobenius norm regularization (Fro-NN).
Revised: Frobenius norm regularization (Fro-NN).

Original: Hinge loss and a regularizer λ ( (cid:107) W (cid:107) F + (cid:107) v (cid:107) 2 ) with λ = 0 .
Revised: Hinge loss and a regularizer λ ( (cid:107) W (cid:107) F + (cid:107) v (cid:107) 2 ) with λ = 0 .

Original: [Figure]
Revised: [Figure]

Original: with λ 
Revised: with λ =

Original: Adversarial training (AT-NN).
Revised: Adversarial training (AT-NN).

Original: Cross-entropy with the adversarial loss against PGD as a regularizer, with the regularization parameter set to 0 .
Revised: Cross-entropy with the adversarial loss against PGD as a regularizer, with the regularization parameter set to 0 .

Original: We found that this regularized loss works better than optimizing only the adversarial loss, which is the defense proposed in Madry et al.
Revised: We found that this regularized loss works better than optimizing only the adversarial loss, which is the defense proposed in Madry et al.

Original: We set the step size of the PGD adversary to 0 .
Revised: We set the step size of the PGD adversary to 0 .

Original: 1 , number of iterations to 40 , and perturbation size to 0 .
Revised: 1 , number of iterations to 40 , and perturbation size to 0 .

Original: Proposed training objective (SDP-NN).
Revised: Proposed training objective (SDP-NN).

Original: Dual SDP objective described in Equation 15 of Section 4.
Revised: Dual SDP objective described in Equation 15 of Section 4.

Original: Implementation details and hyperparameter values are detailed in Section 6.
Revised: Implementation details and hyperparameter values are detailed in Section 6.3.

Original: Evaluating upper bounds.
Revised: Evaluating upper bounds.

Original: Below we will consider various upper bounds on the adversarial loss (cid:96) A opt (based on our method, as well as the Frobenius and spectral bounds described in Section 5).
Revised: Below we will consider various upper bounds on the adversarial loss (cid:96) A opt (based on our method, as well as the Frobenius and spectral bounds described in Section 5).

Original: Ideally we would compare these to the ground-truth adversarial loss (cid:96) A opt , but computing this exactly is difﬁcult.
Revised: Ideally we would compare these to the ground-truth adversarial loss (cid:96) A opt , but computing this exactly is difﬁcult.

Original: Therefore, we compare upper bounds on the adversarial loss with a lower bound on (cid:96) A opt instead.
Revised: Therefore, we compare upper bounds on the adversarial loss with a lower bound on (cid:96) A opt instead.

Original: The loss of any attack provides a valid lower bound and we consider the strong Projected
Revised: The loss of any attack provides a valid lower bound and we consider the strong Projected

Original: Gradient Descent (PGD) attack run against the cross-entropy loss, starting from a random point in
Revised: Gradient Descent (PGD) attack run against the cross-entropy loss, starting from a random point in

Original: B (cid:15) ( x ) , with 5 random restarts.
Revised: B (cid:15) ( x ) , with 5 random restarts.

Original: We observed that PGD against hinge loss did not work well, so we used cross-entropy even for attacking networks trained with the hinge loss.
Revised: We observed that PGD against hinge loss did not work well, so we used cross-entropy even for attacking networks trained with the hinge loss.

Original: 6. Q UALITY OF THE UPPER BOUND
Revised: 6. Q UALITY OF THE UPPER BOUND

Original: For each of the ﬁve networks described above, we computed upper bounds on the 0-1 loss based on our certiﬁcate (which we refer to as the “SDP bound” in this section), as well as the Frobenius and spectral bounds described in Section 5.
Revised: For each of the ﬁve networks described above, we computed upper bounds on the 0-1 loss based on our certiﬁcate (which we refer to as the “SDP bound” in this section), as well as the Frobenius and spectral bounds described in Section 5.

Original: While Section 4 provides a procedure for efﬁciently obtaining an SDP bound as a result of training, for networks not trained with our method we need to solve an SDP at the end of training to obtain certiﬁcates.
Revised: While Section 4 provides a procedure for efﬁciently obtaining an SDP bound as a result of training, for networks not trained with our method we need to solve an SDP at the end of training to obtain certiﬁcates.

Original: Fortunately, this only needs to be done once for every pair of classes.
Revised: Fortunately, this only needs to be done once for every pair of classes.

Original: In our experiments, we use the modeling toolbox YALMIP (Löfberg, 2004) with Sedumi (Sturm, 1999) as a backend to solve the SDPs, using the dual form (14); this took roughly 10 minutes per SDP (around 8 hours in total for a given model).
Revised: In our experiments, we use the modeling toolbox YALMIP (Löfberg, 2004) with Sedumi (Sturm, 1999) as a backend to solve the SDPs, using the dual form (14); this took roughly 10 minutes per SDP (around 8 hours in total for a given model).

Original: In Figure 2, we display average values of the different upper bounds over the 10 , 000 test examples, as well as the corresponding lower bound from PGD.
Revised: In Figure 2, we display average values of the different upper bounds over the 10 , 000 test examples, as well as the corresponding lower bound from PGD.

Original: We ﬁnd that our bound is tighter than the
Revised: We ﬁnd that our bound is tighter than the

Original: Frobenius and spectral bounds for all the networks considered, but its tightness relative to the PGD lower bound varies across the networks.
Revised: Frobenius and spectral bounds for all the networks considered, but its tightness relative to the PGD lower bound varies across the networks.

Original: For instance, our bound is relatively tight on Fro-NN, but unfortunately Fro-NN is not very robust against adversarial examples (the PGD attack exhibits large
Revised: For instance, our bound is relatively tight on Fro-NN, but unfortunately Fro-NN is not very robust against adversarial examples (the PGD attack exhibits large

Original: [Figure]
Revised: [Figure]

Original: error).
Revised: error).

Original: In contrast, the adversarially trained network AT-NN does appear to be robust to attacks, but our certiﬁcate, despite being much tighter than the Frobenius and spectral bounds, is far away from the PGD lower bound.
Revised: In contrast, the adversarially trained network AT-NN does appear to be robust to attacks, but our certiﬁcate, despite being much tighter than the Frobenius and spectral bounds, is far away from the PGD lower bound.

Original: The only network that is both robust and has relatively tight upper bounds is SDP-NN, which was explicitly trained to be both robust and certiﬁable as described in Section 4; we examine this network and the effects of training in more detail in the next subsection.
Revised: The only network that is both robust and has relatively tight upper bounds is SDP-NN, which was explicitly trained to be both robust and certiﬁable as described in Section 4; we examine this network and the effects of training in more detail in the next subsection.

Original: 6. E VALUATING PROPOSED TRAINING OBJECTIVE .
Revised: 6. E VALUATING PROPOSED TRAINING OBJECTIVE .

Original: In the previous section, we saw that the SDP bound, while being tighter than simpler upper bounds, could still be quite loose on arbitrary networks.
Revised: In the previous section, we saw that the SDP bound, while being tighter than simpler upper bounds, could still be quite loose on arbitrary networks.

Original: However, optimizing against the SDP certiﬁcate seemed to make the certiﬁcate tighter.
Revised: However, optimizing against the SDP certiﬁcate seemed to make the certiﬁcate tighter.

Original: In this section, we explore the effect of different optimization objectives in more detail.
Revised: In this section, we explore the effect of different optimization objectives in more detail.

Original: First, we plot on a single axis the best upper bound (i.e., the SDP bound) and the lower bound (from PGD) on the adversarial loss obtained with each of the ﬁve training objectives discussed above.
Revised: First, we plot on a single axis the best upper bound (i.e., the SDP bound) and the lower bound (from PGD) on the adversarial loss obtained with each of the ﬁve training objectives discussed above.

Original: This is given in Figure 3a.
Revised: This is given in Figure 3a.

Original: Neither spectral nor Frobenius norm regularization seems to be helpful for encouraging adversarial robustness—the actual performance of those networks against the PGD attack is worse than the upper bound for SDP-NN against all attacks.
Revised: Neither spectral nor Frobenius norm regularization seems to be helpful for encouraging adversarial robustness—the actual performance of those networks against the PGD attack is worse than the upper bound for SDP-NN against all attacks.

Original: This shows that the SDP certiﬁcate actually provides a useful training objective for encouraging robustness compared to other regularizers.
Revised: This shows that the SDP certiﬁcate actually provides a useful training objective for encouraging robustness compared to other regularizers.

Original: Separately, we can ask whether SDP-NN is robust to actual attacks.
Revised: Separately, we can ask whether SDP-NN is robust to actual attacks.

Original: We explore the robustness of our network in Figure 3b, where we plot the performance of SDP-NN against 3 attacks—the PGD attack from before, the Carlini-Wagner attack (Carlini & Wagner, 2017b)
Revised: We explore the robustness of our network in Figure 3b, where we plot the performance of SDP-NN against 3 attacks—the PGD attack from before, the Carlini-Wagner attack (Carlini & Wagner, 2017b)

Original: (another strong attack), and the weaker Fast Gradient Sign Method (FGSM) baseline.
Revised: (another strong attack), and the weaker Fast Gradient Sign Method (FGSM) baseline.

Original: We see substantial robustness against all 3 attacks, even though our method was not explicitly trained with any of them in mind.
Revised: We see substantial robustness against all 3 attacks, even though our method was not explicitly trained with any of them in mind.

Original: Next, we compare to other bounds reported in the literature.
Revised: Next, we compare to other bounds reported in the literature.

Original: A rough ceiling is given by the network of Madry et al.
Revised: A rough ceiling is given by the network of Madry et al.

Original: (2017), which is a relatively large four-layer convolutional network adversarially trained against PGD.
Revised: (2017), which is a relatively large four-layer convolutional network adversarially trained against PGD.

Original: While this network has no accompanying certiﬁcate of robustness, it was evaluated against a number of attack strategies and had worst-case error 11% at (cid:15) = 0 .
Revised: While this network has no accompanying certiﬁcate of robustness, it was evaluated against a number of attack strategies and had worst-case error 11% at (cid:15) = 0 .

Original: Another set of numbers comes from Carlini et al.
Revised: Another set of numbers comes from Carlini et al.

Original: (2017), who use formal veriﬁcation methods to compute A opt exactly on 10 input examples for a small ( 72 -node) variant of the Madry et al.
Revised: (2017), who use formal veriﬁcation methods to compute A opt exactly on 10 input examples for a small ( 72 -node) variant of the Madry et al.

Original: network.
Revised: network.

Original: The authors reported to us that this network misclassiﬁes 6 out of 10 examples at (cid:15) = 0 .
Revised: The authors reported to us that this network misclassiﬁes 6 out of 10 examples at (cid:15) = 0 .

Original: 05 (we note that 4 out of 10 of these were misclassiﬁed to start with, but 3 of the 4 can also be ﬂipped to a different wrong class with some (cid:15) < 0 . 07 ).
Revised: 05 (we note that 4 out of 10 of these were misclassiﬁed to start with, but 3 of the 4 can also be ﬂipped to a different wrong class with some (cid:15) < 0 . 07 ).

Original: At the value (cid:15) = 0 . 1 for which it was tuned, SDP-NN has error 16% against the PGD attack, and an upper bound of 35% error against any attack.
Revised: At the value (cid:15) = 0 . 1 for which it was tuned, SDP-NN has error 16% against the PGD attack, and an upper bound of 35% error against any attack.

Original: This is substantially better than the small 72 -node network, but also much worse than the full Madry et al.
Revised: This is substantially better than the small 72 -node network, but also much worse than the full Madry et al.

Original: network.
Revised: network.

Original: How much of the latter looseness comes from conservatism in our method, versus the fact that our network has only two layers?
Revised: How much of the latter looseness comes from conservatism in our method, versus the fact that our network has only two layers?

Original: We can get some idea by considering the AT-NN network, which was trained similarly to Madry et al., but uses the same architecture as SDP-NN.
Revised: We can get some idea by considering the AT-NN network, which was trained similarly to Madry et al., but uses the same architecture as SDP-NN.

Original: From Figure 3a, we see that the error of SDP-NN against
Revised: From Figure 3a, we see that the error of SDP-NN against

Original: PGD ( 16% ) is not much worse than that of AT-NN ( 11% ), even though AT-NN was explicitly trained against the PGD attack.
Revised: PGD ( 16% ) is not much worse than that of AT-NN ( 11% ), even though AT-NN was explicitly trained against the PGD attack.

Original: This suggests that most of the gap comes from the smaller network depth,
Revised: This suggests that most of the gap comes from the smaller network depth,

Original: [Figure]
Revised: [Table]

Original: rather than from conservatism in the SDP bound.
Revised: rather than from conservatism in the SDP bound.

Original: We are currently in the process of extending our approach to deeper networks, and optimistic about obtaining improved bounds with such networks.
Revised: We are currently in the process of extending our approach to deeper networks, and optimistic about obtaining improved bounds with such networks.

Revised: Finally, we compare with the approach proposed in Kolter & Wong (2017) whose work appeared shortly after an initial version of our paper.

Revised: They provide an upper bound on the adversarial loss using linear programs (LP) followed by a method to efﬁciently train networks to minimize this upper bound.

Revised: In order to compare with SDP-NN, the authors provided us with a network with the same architecture as SDP-NN, but trained using their LP based objective.

Revised: We call this network LP-NN.

Revised: Table 1 shows that LP-NN and SDP-NN are comparable in terms of their robustness against PGD, and the robustness guarantees that they come with.

Original:  6. I MPLEMENTATION DETAILS
Revised: Interestingly, the SDP and LP approaches provide vacuous bounds for networks not trained to minimize the respective upper bounds (though these networks are indeed robust). This suggests that these two approaches are comparable, but complementary. Finally, we note that in contrast to this work, the approach of Kolter & Wong (2017) extends to deeper networks, which allows them to train a four-layer CNN with a provable upper bound on adversarial error of 8 . 4% error. 6. I MPLEMENTATION DETAILS

Original: We implemented our training objective in TensorFlow, and implemented λ + max as a custom operator using SciPy’s implementation of the Lanczos algorithm for fast top eigenvector computation; occasionally Lanczos fails to converge due to a small eigen-gap, in which case we back off to a full
Revised: We implemented our training objective in TensorFlow, and implemented λ + max as a custom operator using SciPy’s implementation of the Lanczos algorithm for fast top eigenvector computation; occasionally Lanczos fails to converge due to a small eigen-gap, in which case we back off to a full

Original: SVD.
Revised: SVD.

Original: We used hinge loss as the classiﬁcation loss, and decayed the learning rate in steps from 10 −to 10 − 5 , decreasing by a factor of 10 every 30 epochs.
Revised: We used hinge loss as the classiﬁcation loss, and decayed the learning rate in steps from 10 −to 10 − 5 , decreasing by a factor of 10 every 30 epochs.

Original: Each gradient step involves computing top eigenvectors for 45 different matrices, one for each pair of classes ( i, j ) .
Revised: Each gradient step involves computing top eigenvectors for 45 different matrices, one for each pair of classes ( i, j ) .

Original: In order to speed up computation, for each update, we randomly pick i t and only compute gradients for pairs ( i t , j ) , j (cid:54) =
Revised: In order to speed up computation, for each update, we randomly pick i t and only compute gradients for pairs ( i t , j ) , j (cid:54) =

Original: i t , requiring only 9 top eigenvector computations in each step.
Revised: i t , requiring only 9 top eigenvector computations in each step.

Original: For the regularization parameters λ ij , the simplest idea is to set them all equal to the same value; this leads to the unweighted regularization scheme where λ ij = λ for all pairs ( i, j ) .
Revised: For the regularization parameters λ ij , the simplest idea is to set them all equal to the same value; this leads to the unweighted regularization scheme where λ ij = λ for all pairs ( i, j ) .

Original: We tuned λ to 0 .
Revised: We tuned λ to 0 .

Original: 05 , which led to reasonably good bounds.
Revised: 05 , which led to reasonably good bounds.

Original: However, we observed that certain pairs of classes tended to have larger margins f ij ( x ) than other classes, which meant that certain label pairs appeared in the maximum of (12) much more often.
Revised: However, we observed that certain pairs of classes tended to have larger margins f ij ( x ) than other classes, which meant that certain label pairs appeared in the maximum of (12) much more often.

Original: That led us to consider a weighted regularization scheme with λ ij = w ij λ , where w ij is the fraction of training points for which the the label i (or j ) appears as the maximizing term in (12).
Revised: That led us to consider a weighted regularization scheme with λ ij = w ij λ , where w ij is the fraction of training points for which the the label i (or j ) appears as the maximizing term in (12).

Original: We updated the values of these weights every 20 epochs.
Revised: We updated the values of these weights every 20 epochs.

Original: Figure 4a compares the PGD lower bound and SDP upper bound for the unweighted and weighted networks.
Revised: Figure 4a compares the PGD lower bound and SDP upper bound for the unweighted and weighted networks.

Original: The weighted network is better than the unweighted network for both the lower and upper bounds.
Revised: The weighted network is better than the unweighted network for both the lower and upper bounds.

Original: Finally, we saw in Equation 16 of Section 4 that the dual variables c ij provide a quick-to-compute certiﬁcate of robustness.
Revised: Finally, we saw in Equation 16 of Section 4 that the dual variables c ij provide a quick-to-compute certiﬁcate of robustness.

Original: Figure 4b shows that the certiﬁcates provided by these dual variables are very close to what we would obtain by fully optimizing the semideﬁnite programs.
Revised: Figure 4b shows that the certiﬁcates provided by these dual variables are very close to what we would obtain by fully optimizing the semideﬁnite programs.

Original: These dual certiﬁcates made it easy to track robustness across epochs of training and to tune hyperparameters.
Revised: These dual certiﬁcates made it easy to track robustness across epochs of training and to tune hyperparameters.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: In this work, we proposed a method for producing certiﬁcates of robustness for neural networks, and for training against these certiﬁcates to obtain networks that are provably robust against adversaries.
Revised: In this work, we proposed a method for producing certiﬁcates of robustness for neural networks, and for training against these certiﬁcates to obtain networks that are provably robust against adversaries.

Original: Related work.
Revised: Related work.

Revised: In parallel and independent work, Kolter & Wong (2017) also provide provably robust networks against (cid:96) ∞ perturbations by using convex relaxations.

Revised: While our approach uses a single semideﬁnite program to compute an upper bound on the adversarial loss, Kolter & Wong

Revised: [Figure]

Original:  Katz et al.
Revised: (2017) use separate linear programs for every data point, and apply their method to networks of depth up to four. In theory, neither bound is strictly tighter than the other, and our experiments (Table 1) suggest that the two bounds are complementary. Combining the approaches seems to be a promising future direction. Katz et al.

Original: (2017a)
Revised: (2017a)

Original: and the follow-up Carlini et al.
Revised: and the follow-up Carlini et al.

Original: (2017) also provide certiﬁcates of robustness for neural networks against (cid:96) ∞ perturbations.
Revised: (2017) also provide certiﬁcates of robustness for neural networks against (cid:96) ∞ perturbations.

Original: That work uses SMT solvers, which are a tool from the formal veriﬁcation literature.
Revised: That work uses SMT solvers, which are a tool from the formal veriﬁcation literature.

Original: The SMT solver can answer the binary question “Is there an adversarial example within distance (cid:15) of the input x ?” , and is correct whenever it terminates.
Revised: The SMT solver can answer the binary question “Is there an adversarial example within distance (cid:15) of the input x ?” , and is correct whenever it terminates.

Original: The main drawback of SMT and similar formal veriﬁcation methods is that they are slow—they have worst-case exponential-time scaling in the size of the network; moreover, to use them during training would require a separate search for each gradient step.
Revised: The main drawback of SMT and similar formal veriﬁcation methods is that they are slow—they have worst-case exponential-time scaling in the size of the network; moreover, to use them during training would require a separate search for each gradient step.

Original: Huang et al.
Revised: Huang et al.

Original: (2017) use SMT solvers and are able to analyze state-of-the-art networks on MNIST, but they make various approximations such that their numbers are not true upper bounds.
Revised: (2017) use SMT solvers and are able to analyze state-of-the-art networks on MNIST, but they make various approximations such that their numbers are not true upper bounds.

Original: Bastani et al.
Revised: Bastani et al.

Original: (2016) provide tractable certiﬁcates but require (cid:15) to be small enough to ensure that the entire (cid:96) ∞ ball around an input lies within the same linear region.
Revised: (2016) provide tractable certiﬁcates but require (cid:15) to be small enough to ensure that the entire (cid:96) ∞ ball around an input lies within the same linear region.

Original: For the networks and values of (cid:15) that we consider in our paper, we found that this condition did not hold.
Revised: For the networks and values of (cid:15) that we consider in our paper, we found that this condition did not hold.

Original: Recently, Hein & Andriushchenko (2017) proposed a bound for guaranteeing robustness to (cid:96) p -norm perturbations, based on the maximum pp − 1 -norm of the gradient in the (cid:15) -ball around the inputs.
Revised: Recently, Hein & Andriushchenko (2017) proposed a bound for guaranteeing robustness to (cid:96) p -norm perturbations, based on the maximum pp − 1 -norm of the gradient in the (cid:15) -ball around the inputs.

Original: Hein & Andriushchenko (2017) show how to efﬁciently compute this bound for p
Revised: Hein & Andriushchenko (2017) show how to efﬁciently compute this bound for p

Original: = 2 , as opposed to our work which focuses on (cid:96) ∞ and requires different techniques to achieve scalability.
Revised: = 2 , as opposed to our work which focuses on (cid:96) ∞ and requires different techniques to achieve scalability.

Original: Madry et al.
Revised: Madry et al.

Original: (2017) perform adversarial training against PGD on the MNIST and CIFAR-10 datasets, obtaining networks that they suggest are “secure against ﬁrst-order adversaries”.
Revised: (2017) perform adversarial training against PGD on the MNIST and CIFAR-10 datasets, obtaining networks that they suggest are “secure against ﬁrst-order adversaries”.

Original: However, this is based on an empirical observation that PGD is nearly-optimal among gradient-based attacks, and does not correspond to any formal robustness guarantee.
Revised: However, this is based on an empirical observation that PGD is nearly-optimal among gradient-based attacks, and does not correspond to any formal robustness guarantee.

Original: Finally, the notion of a certiﬁcate appears in the theory of convex optimization, but means something different in that context; speciﬁcally, it corresponds to a proof that a point is near the optimum of a convex function, whereas here our certiﬁcates provide upper bounds on non-convex functions.
Revised: Finally, the notion of a certiﬁcate appears in the theory of convex optimization, but means something different in that context; speciﬁcally, it corresponds to a proof that a point is near the optimum of a convex function, whereas here our certiﬁcates provide upper bounds on non-convex functions.

Original: Additionally, while robust optimization (Bertsimas et al., 2011) provides a tool for optimizing objectives with robustness constraints, applying it directly would involve the same intractable optimization for
Revised: Additionally, while robust optimization (Bertsimas et al., 2011) provides a tool for optimizing objectives with robustness constraints, applying it directly would involve the same intractable optimization for

Original: A opt that we deal with here.
Revised: A opt that we deal with here.

Original: Other approaches to veriﬁcation.
Revised: Other approaches to veriﬁcation.

Original: While they have not been explored in the context of neural networks, there are approaches in the control theory literature for verifying robustness of dynamical systems, based on Lyapunov functions (Lyapunov, 1892; 1992).
Revised: While they have not been explored in the context of neural networks, there are approaches in the control theory literature for verifying robustness of dynamical systems, based on Lyapunov functions (Lyapunov, 1892; 1992).

Original: We can think of the activations in a neural network as the evolution of a time-varying dynamical system, and attempt to prove stability around a trajectory of this system (Tedrake et al., 2010; Tobenkin et al., 2011).
Revised: We can think of the activations in a neural network as the evolution of a time-varying dynamical system, and attempt to prove stability around a trajectory of this system (Tedrake et al., 2010; Tobenkin et al., 2011).

Original: Such methods typically use sum-of-squares veriﬁcation (Papachristodoulou & Prajna, 2002; 2005; Parrilo, 2003) and are restricted to relatively low-dimensional dynamical systems, but could plausibly scale to larger settings.
Revised: Such methods typically use sum-of-squares veriﬁcation (Papachristodoulou & Prajna, 2002; 2005; Parrilo, 2003) and are restricted to relatively low-dimensional dynamical systems, but could plausibly scale to larger settings.

Original: Another approach is to construct families of networks that are provably robust a priori, which would remove the need to verify robustness of the learned model; to our knowledge this has not been done for any expressive model families.
Revised: Another approach is to construct families of networks that are provably robust a priori, which would remove the need to verify robustness of the learned model; to our knowledge this has not been done for any expressive model families.

Original: Adversarial examples and secure ML.
Revised: Adversarial examples and secure ML.

Original: There has been a great deal of recent work on the security of
Revised: There has been a great deal of recent work on the security of

Original: ML systems; we provide only a sampling here, and refer the reader to Barreno et al.
Revised: ML systems; we provide only a sampling here, and refer the reader to Barreno et al.

Original: (2010), Biggio et al.
Revised: (2010), Biggio et al.

Original: (2014a), Papernot et al.
Revised: (2014a), Papernot et al.

Original: (2016b), and Gardiner & Nagaraja (2016)
Revised: (2016b), and Gardiner & Nagaraja (2016)

Original: for some recent surveys.
Revised: for some recent surveys.

Original: Adversarial examples for neural networks were ﬁrst discovered by Szegedy et al.
Revised: Adversarial examples for neural networks were ﬁrst discovered by Szegedy et al.

Original: (2014), and since then a number of attacks and defenses have been proposed.
Revised: (2014), and since then a number of attacks and defenses have been proposed.

Original: We have already discussed gradientbased methods as well as defenses based on adversarial training.
Revised: We have already discussed gradientbased methods as well as defenses based on adversarial training.

Original: There are also other attacks based on, e.g., saliency maps (Papernot et al., 2016a), KL divergence (Miyato et al., 2015), and elastic net optimization (Chen et al., 2017); many of these attacks are collated in the cleverhans repository (Goodfellow et al., 2016).
Revised: There are also other attacks based on, e.g., saliency maps (Papernot et al., 2016a), KL divergence (Miyato et al., 2015), and elastic net optimization (Chen et al., 2017); many of these attacks are collated in the cleverhans repository (Goodfellow et al., 2016).

Original: For defense, rather than making networks robust to adversaries, some work has focused on simply detecting adversarial examples.
Revised: For defense, rather than making networks robust to adversaries, some work has focused on simply detecting adversarial examples.

Original: However, Carlini & Wagner (2017a) recently showed that essentially all known detection methods can be subverted by strong attacks.
Revised: However, Carlini & Wagner (2017a) recently showed that essentially all known detection methods can be subverted by strong attacks.

Original: As explained in Barreno et al.
Revised: As explained in Barreno et al.

Original: (2010), there are a number of different attack models beyond the testtime attacks considered here, based on different attacker goals and capabilities.
Revised: (2010), there are a number of different attack models beyond the testtime attacks considered here, based on different attacker goals and capabilities.

Original: For instance, one can consider data poisoning attacks, where an attacker modiﬁes the training set in an effort to affect test-time performance.
Revised: For instance, one can consider data poisoning attacks, where an attacker modiﬁes the training set in an effort to affect test-time performance.

Original: Newsome et al.
Revised: Newsome et al.

Original: (2006), Laskov & Šrndi`c (2014), and Biggio et al.
Revised: (2006), Laskov & Šrndi ` c (2014), and Biggio et al.

Original: (2014b) have demonstrated poisoning attacks against real-world systems.
Revised: (2014b) have demonstrated poisoning attacks against real-world systems.

Original: Other types of certiﬁcates.
Revised: Other types of certiﬁcates.

Original: Certiﬁcates of performance for machine learning systems are desirable in a number of settings.
Revised: Certiﬁcates of performance for machine learning systems are desirable in a number of settings.

Original: This includes verifying safety properties of air trafﬁc control systems (Katz et al., 2017a;b) and self-driving cars (O’Kelly et al., 2016; 2017), as well as security applications such as robustness to training time attacks (Steinhardt et al., 2017).
Revised: This includes verifying safety properties of air trafﬁc control systems (Katz et al., 2017a;b) and self-driving cars (O’Kelly et al., 2016; 2017), as well as security applications such as robustness to training time attacks (Steinhardt et al., 2017).

Original: More broadly, certiﬁcates of performance are likely necessary for deploying machine learning systems in critical infrastructure such as internet packet routing (Winstein & Balakrishnan, 2013; Sivaraman et al., 2014).
Revised: More broadly, certiﬁcates of performance are likely necessary for deploying machine learning systems in critical infrastructure such as internet packet routing (Winstein & Balakrishnan, 2013; Sivaraman et al., 2014).

Original: In robotics, certiﬁcates of stability are routinely used both for safety veriﬁcation (Lygeros et al., 1999; Mitchell et al., 2005) and controller synthesis (Ba¸sar & Bernhard, 2008; Tedrake et al., 2010).
Revised: In robotics, certiﬁcates of stability are routinely used both for safety veriﬁcation (Lygeros et al., 1999; Mitchell et al., 2005) and controller synthesis (Ba¸sar & Bernhard, 2008; Tedrake et al., 2010).

Original: In traditional veriﬁcation work, Rice’s theorem (Rice, 1953) is a strong impossibility result essentially stating that most properties of most programs are undecidable.
Revised: In traditional veriﬁcation work, Rice’s theorem (Rice, 1953) is a strong impossibility result essentially stating that most properties of most programs are undecidable.

Original: Similarly, we should expect that verifying robustness for arbitrary neural networks is hard.
Revised: Similarly, we should expect that verifying robustness for arbitrary neural networks is hard.

Original: However, the results in this work suggest that it is possible to learn neural networks that are amenable to veriﬁcation, in the same way that it is possible to write programs that can be formally veriﬁed.
Revised: However, the results in this work suggest that it is possible to learn neural networks that are amenable to veriﬁcation, in the same way that it is possible to write programs that can be formally veriﬁed.

Original: Optimistically, given expressive enough certiﬁcation methods and model families, as well as strong enough speciﬁcations of robustness, one could even hope to train vector representations of natural images with strong robustness properties, thus ﬁnally closing the chapter on adversarial vulnerabilities in the visual domain
Revised: Optimistically, given expressive enough certiﬁcation methods and model families, as well as strong enough speciﬁcations of robustness, one could even hope to train vector representations of natural images with strong robustness properties, thus ﬁnally closing the chapter on adversarial vulnerabilities in the visual domain. Reproducibility.

Revised: All code, data and experiments for this paper are available on the

Original: A D UALITY
Revised: Codalab platform at https://worksheets.codalab.org/worksheets/ 0xa21e794020bb474d8804ec7bc0543f52/ . Acknowledgements. This work was partially supported by a Future of Life Institute Research Award and Open Philanthrophy Project Award.

Revised: JS was supported by a Fannie & John Hertz Foundation Fellowship and an NSF Graduate Research Fellowship.

Revised: We are also grateful to Guy Katz,

Revised: Zico Kolter and Eric Wong for providing relevant experimental results for comparison, as well as to the anonymous reviewers for useful feedback and references.

Revised: A D UALITY

Original: In this section we justify the duality relation (14).
Revised: In this section we justify the duality relation (14).

Original: Recall that the primal program is maximize (cid:104) M, P (cid:105) (19) subject to P (cid:23) 0 , diag( P ) ≤ 1 .
Revised: Recall that the primal program is maximize (cid:104) M, P (cid:105) (19) subject to P (cid:23) 0 , diag( P ) ≤ 1 .

Original: Rather than taking the dual directly, we ﬁrst add the redundant constraint tr( P ) ≤ d + m + 1 (it is redundant because the SDP is in d + m + 1 dimensions and diag( P ) ≤ 1 ).
Revised: Rather than taking the dual directly, we ﬁrst add the redundant constraint tr( P ) ≤ d + m + 1 (it is redundant because the SDP is in d + m + 1 dimensions and diag( P ) ≤ 1 ).

Original: This yields maximize (cid:104)M
Revised: This yields maximize (cid:104)

Original: [Equation]
Revised: [Equation]

Original:  P
Revised: (20) subject to P (cid:23) 0 , diag( P

Original: [Equation]
Revised: [Equation]

Original: (20) subject to P (cid:23) 0 , diag( P ) ≤ 1 , tr( P ) ≤ d + m + 1 .
Revised: ≤ 1 , tr( P ) ≤ d + m + 1 .

Original: We now form the Lagrangian for the constraints diag( P ) ≤ 1 , leaving the other two constraints as-is.
Revised: We now form the Lagrangian for the constraints diag( P ) ≤ 1 , leaving the other two constraints as-is.

Original: This yields the equivalent optimization problem 
Revised: This yields the equivalent optimization problem maximize

Original: [Equation]
Revised: [Equation]

Original: 21) subject to P (cid:23) 
Revised: (21) subject to P (cid:23) 0 , tr

Original: [Equation]
Revised: [Equation]

Original: tr
Revised: P

Original: [Equation]
Revised: [Equation]

Original:  Now, we apply minimax duality to swap the order of min and max; the value of (21) is thus equal to minimize
Revised: ≤ d + m + 1 . Now, we apply minimax duality to swap the order of min and max; the value of (21) is thus equal to minimize

Original: [Equation]
Revised: [Equation]

Revised: P

Original: [Equation]
Revised: [Equation]

Original:  subject to c ≥ 0 .
Revised: (22) subject to c ≥ 0 .

Original: The inner maximum can be simpliﬁed as
Revised: The inner maximum can be simpliﬁed as

Original: [Equation]
Revised: [Equation]

Original: Therefore, (22) simpliﬁes to
Revised: Therefore, (22) simpliﬁes to

Original: [Equation]
Revised: [Equation]

Revised: to

Original: [Equation]
Revised: [Equation]

Original: subject to c ≥ 0 . This is almost the form given in (14), except that c is constrained to be non-negative and we have 1 (cid:62) c instead of 1 (cid:62) max( c, 0) .
Revised: This is almost the form given in (14), except that c is constrained to be non-negative and we have 1 (cid:62) c instead of 1 (cid:62) max( c, 0) .

Original: However, note that for the λ +max term, it is always better for c to be larger; therefore, replacing c with max( c, 0) means that the optimal value of c will always be nonnegative, thus allowing us to drop the c ≥ 0 constraint and optimize c in an unconstrained manner.
Revised: However, note that for the λ +max term, it is always better for c to be larger; therefore, replacing c with max( c, 0) means that the optimal value of c will always be nonnegative, thus allowing us to drop the c ≥ 0 constraint and optimize c in an unconstrained manner.

Original: This ﬁnally yields the claimed duality relation (14).
Revised: This ﬁnally yields the claimed duality relation (14).