Original: A DVERSARIAL T RAINING AND P ROVABLE D EFENSES : B RIDGING THE G AP
Revised: A DVERSARIAL T RAINING AND P ROVABLE D EFENSES : B RIDGING THE G AP

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We propose a new method to train neural networks based on a novel combination of adversarial training and provable defenses.
Revised: We propose a new method to train neural networks based on a novel combination of adversarial training and provable defenses.

Original: The key idea is to model training as a procedure which includes both , the veriﬁer and the adversary.
Revised: The key idea is to model training as a procedure which includes both , the veriﬁer and the adversary.

Original: In every iteration, the veriﬁer aims to certify the network using convex relaxation while the adversary tries to ﬁnd inputs inside that convex relaxation which cause veriﬁcation to fail.
Revised: In every iteration, the veriﬁer aims to certify the network using convex relaxation while the adversary tries to ﬁnd inputs inside that convex relaxation which cause veriﬁcation to fail.

Original: We experimentally show that this training method is promising and achieves the best of both worlds – it produces amodel with state-of-the-art accuracy (74.8%) and certiﬁed robustness (55.9%) on the challenging CIFARdataset with a 2/255 L ∞ perturbation.
Revised: We experimentally show that this training method is promising and achieves the best of both worlds – it produces a state-of-the-art neural network with certiﬁed robustness of 58.1% and accuracy of 78.8% on the challenging CIFARdataset with a 2/255 L ∞ perturbation.

Original: This is a signiﬁcant improvement over the currently known best results of68.3% accuracy and 53.9% certiﬁed robustness, achieved using a 5 times larger network than our work.
Revised: This signiﬁcantly improves over the currently known best results of 53.9% certiﬁed robustness and 68.3% accuracy.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The discovery of adversarial examples in deep learning (Szegedy et al., 2013) has increased the importance of creating new training methods which produce accurate and robust neural networks with provable guarantees.
Revised: The discovery of adversarial examples in deep learning (Szegedy et al., 2013) has increased the importance of creating new training methods which produce accurate and robust neural networks with provable guarantees.

Original: Existing work: adversarial and provable defenses Adversarial training (Goodfellow et al., 2015; Kurakin et al., 2017) provides a basis framework which augments the training procedure with adversarial inputs produced by an adversarial attack.
Revised: Existing work: adversarial and provable defenses Adversarial training (Goodfellow et al., 2015; Kurakin et al., 2017) provides a basis framework which augments the training procedure with adversarial inputs produced by an adversarial attack.

Original: Madry et al.
Revised: Madry et al.

Original: (2018) instantiated adversarial training using a strong iterative adversary and showed that their approach can train models which are highly robust against the strongest known adversarial attacks (Carlini & Wagner, 2017).
Revised: (2018) instantiated adversarial training using a strong iterative adversary and showed that their approach can train models which are highly robust against the strongest known adversarial attacks (Carlini & Wagner, 2017).

Original: This method has also been used to train robust ImageNet models (Xie et al., 2019).
Revised: This method has also been used to train robust ImageNet models (Xie et al., 2019).

Original: While promising, the main drawback of the method is that when instantiated in practice, via an approximation of an otherwise intractable optimization problem, it provides no guarantees – it does not produce a certiﬁcate that there are no possible adversarial attacks which could potentially break the model.
Revised: While promising, the main drawback of the method is that when instantiated in practice, via an approximation of an otherwise intractable optimization problem, it provides no guarantees – it does not produce a certiﬁcate that there are no possible adversarial attacks which could potentially break the model.

Original: To address this lack of guarantees, recent line of work on provable defenses (Wong & Kolter, 2018;
Revised: To address this lack of guarantees, recent line of work on provable defenses (Wong & Kolter, 2018;

Original: Raghunathan et al., 2018; Mirman et al., 2018) has proposed to train neural networks which are certiﬁably robust under a speciﬁc attacker threat model.
Revised: Raghunathan et al., 2018; Mirman et al., 2018) has proposed to train neural networks which are certiﬁably robust under a speciﬁc attacker threat model.

Original: However, these guarantees come at the cost of a signiﬁcantly lower standard accuracy than models trained using adversarial training.
Revised: However, these guarantees come at the cost of a signiﬁcantly lower standard accuracy than models trained using adversarial training.

Original: This setting raises a natural question – can we leverage ideas from both, adversarial training techniques and provable defense methods, so to obtain models with high accuracy and certiﬁed robustness?
Revised: This setting raises a natural question – can we leverage ideas from both, adversarial training techniques and provable defense methods, so to obtain models with high accuracy and certiﬁed robustness?

Original: This work: combining adversarial and provable defenses In this work, we take a step towards addressing this challenge.
Revised: This work: combining adversarial and provable defenses In this work, we take a step towards addressing this challenge.

Original: We show that it is possible to train more accurate and provably robust neural networks using the same convex relaxations as those used in existing, state-of-the-art provable defense methods, but with a new, different optimization procedure inspired by adversarial training.
Revised: We show that it is possible to train more accurate and provably robust neural networks using the same convex relaxations as those used in existing, state-of-the-art provable defense methods, but with a new, different optimization procedure inspired by adversarial training.

Original: Our optimization works as follows: (i) to certify a property (e.g., robustness) of the network, the veriﬁer produces a convex relaxation of all possible intermediate vector outputs in the neural network, then (ii) an adversary now searches over this (intermediate) convex regions in order to ﬁnd, what we refer to as a latent adversarial example – a concrete intermediate input contained in the relaxation that when propagated through the network causes a misclassiﬁcation that prevents veriﬁcation, and ﬁnally (iii) the resulting latent adversarial examples are now incorporated into our training scheme using adversarial training.
Revised: Our optimization works as follows: (i) to certify a property (e.g., robustness) of the network, the veriﬁer produces a convex relaxation of all possible intermediate vector outputs in the neural network, then (ii) an adversary now searches over this (intermediate) convex regions in order to ﬁnd, what we refer to as a latent adversarial example – a concrete intermediate input contained in the relaxation that when propagated through the network causes a misclassiﬁcation that prevents veriﬁcation, and ﬁnally (iii) the resulting latent adversarial examples are now incorporated into our training scheme using adversarial training.

Original: Overall, we can see this method as bridging the gap between adversarial training and provable defenses (it can conceptually be instantiated with any convex relaxation).
Revised: Overall, we can see this method as bridging the gap between adversarial training and provable defenses (it can conceptually be instantiated with any convex relaxation).

Original: We experimentally show that the method is promising and results in a neural network with state-ofthe-art 74.8% accuracy and 55.9% certiﬁed robustness on the challenging CIFAR-10 dataset with 2/255 L ∞ perturbation (the best known existing results are 68.3% accuracy and 53.9% certiﬁed robustness using 5 times larger network Wong et al.
Revised: We experimentally show that the method is promising and results in a neural network with state-of-theart 78.8% accuracy and 58.1% certiﬁed robustness on the challenging CIFAR-10 dataset with 2/ L ∞ perturbation (the best known existing results are 68.3% accuracy and 53.9% certiﬁed robustness using 5 times larger network Wong et al.

Original: Main Contributions
Revised: Main Contributions

Original: Our key contributions are:
Revised: Our key contributions are:

Original: • A new method we refer to as layerwise adversarial training which can train provably robust neural networks and conceptually bridges the gap between adversarial training and existing provable defense methods.
Revised: • A new method we refer to as layerwise adversarial training which can train provably robust neural networks and conceptually bridges the gap between adversarial training and existing provable defense methods.

Original: • Instantiation of layerwise adversarial training using linear convex relaxations used in prior work (accomplished by introducing a projection operator).
Revised: • Instantiation of layerwise adversarial training using linear convex relaxations used in prior work (accomplished by introducing a projection operator).

Original: • Experimental results showing layerwise adversarial training can train neural network models which achieve both, state-of-the-art accuracy and certiﬁed robustness on CIFAR-10 with 2/255 L ∞ perturbation.
Revised: • Experimental results showing layerwise adversarial training can train neural network models which achieve both, state-of-the-art accuracy and certiﬁed robustness on CIFAR-10 with 2/255 L ∞ perturbation.

Original: Overall, we believe the method presented in this work is a promising step towards training models that enjoy both, higher accuracy and higher certiﬁcation guarantees.
Revised: Overall, we believe the method presented in this work is a promising step towards training models that enjoy both, higher accuracy and higher certiﬁcation guarantees.

Original: An interesting item for future work would be to explore instantiations of the method with other convex relaxations than the one considered here.
Revised: An interesting item for future work would be to explore instantiations of the method with other convex relaxations than the one considered here.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: We now discuss some of the closely related work on robustness of neural networks.
Revised: We now discuss some of the closely related work on robustness of neural networks.

Original: Heuristic adversarial defenses After the ﬁrst introduction of adversarial examples (Szegedy et al., 2013; Biggio et al., 2013), defense mechanisms to train robust neural networks were built based on the inclusion of adversarial examples to the training set (Kurakin et al., 2017; Goodfellow et al., 2015).
Revised: Heuristic adversarial defenses After the ﬁrst introduction of adversarial examples (Szegedy et al., 2013; Biggio et al., 2013), defense mechanisms to train robust neural networks were built based on the inclusion of adversarial examples to the training set (Kurakin et al., 2017; Goodfellow et al., 2015).

Original: Models trained using adversarial training with projected gradient descent (PGD) (Madry et al., 2018) were shown to be robust against the strongest known attacks (Carlini & Wagner, 2017).
Revised: Models trained using adversarial training with projected gradient descent (PGD) (Madry et al., 2018) were shown to be robust against the strongest known attacks (Carlini & Wagner, 2017).

Original: This is in contrast to other defense mechanisms which have been broken by new attack techniques (Athalye et al., 2018).
Revised: This is in contrast to other defense mechanisms which have been broken by new attack techniques (Athalye et al., 2018).

Original: While models trained using adversarial training achieve robustness against strong adversaries, there are no guarantees that model is robust against any kind of adversarial attack under the threat model considered.
Revised: While models trained using adversarial training achieve robustness against strong adversaries, there are no guarantees that model is robust against any kind of adversarial attack under the threat model considered.

Original: Provable adversarial defenses Another line of work proposes to learn classiﬁers which come with robustness guarantees.
Revised: Provable adversarial defenses Another line of work proposes to learn classiﬁers which come with robustness guarantees.

Original: These approaches are based on linear (Wong & Kolter, 2018) or semideﬁnite (Raghunathan et al., 2018;
Revised: These approaches are based on linear (Wong & Kolter, 2018) or semideﬁnite (Raghunathan et al., 2018;

Original: Dvijotham et al., 2018a) relaxations, hybrid zonotope (Mirman et al., 2018) or interval bound propagation (Gowal et al., 2018)
Revised: Dvijotham et al., 2018a) relaxations, hybrid zonotope (Mirman et al., 2018) or interval bound propagation (Gowal et al., 2018)

Original: While these approaches currently obtain robustness guarantees, accuracy of these networks is relatively small and limits practical use of these methods.
Revised: While these approaches currently obtain robustness guarantees, accuracy of these networks is relatively small and limits practical use of these methods.

Original: There has also been recent work on certiﬁcation of general neural networks, not necessarily trained in a special way.
Revised: There has also been recent work on certiﬁcation of general neural networks, not necessarily trained in a special way.

Original: These methods are based on SMT solvers (Katz et al., 2017), mixed-integer linear programs (Tjeng et al., 2019), abstract interpretation (Gehr et al., 2018), restricted polyhedra (Singh et al., 2019b) or combinations of those (Singh et al., 2019a).
Revised: These methods are based on SMT solvers (Katz et al., 2017), mixed-integer linear programs (Tjeng et al., 2019), abstract interpretation (Gehr et al., 2018), restricted polyhedra (Singh et al., 2019b) or combinations of those (Singh et al., 2019a).

Original: Another line of work proposes to replace neural networks with a randomized classiﬁer (Lecuyer et al., 2018;
Revised: Another line of work proposes to replace neural networks with a randomized classiﬁer (Lecuyer et al., 2018;

Original: Cohen et al., 2019;
Revised: Cohen et al., 2019;

Original: Salman et al., 2019a) which comes with probabilistic guarantees on its robustness.
Revised: Salman et al., 2019a) which comes with probabilistic guarantees on its robustness.

Original: While these approaches scale to larger datasets such as ImageNet (although with probabilistc instead of exact guarantees), their bounds come from the relationship between L 2 robustness and Gaussian distribution.
Revised: While these approaches scale to larger datasets such as ImageNet (although with probabilistc instead of exact guarantees), their bounds come from the relationship between L 2 robustness and Gaussian distribution.

Original: In this paper, we consider general veriﬁcation problem (Qin et al., 2019) where input is not necessarily limited to an L p ball, but arbitrary convex set.
Revised: In this paper, we consider general veriﬁcation problem (Qin et al., 2019) where input is not necessarily limited to an L p ball, but arbitrary convex set.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this work we consider a threat model where an adversary is allowed to transform an input x ∈ R dinto any point from a convex set S 0 ( x ) ⊆
Revised: In this work we consider a threat model where an adversary is allowed to transform an input x ∈ R dinto any point from a convex set S 0 ( x ) ⊆

Original: R d 0 .
Revised: R d 0 .

Original: For example, for a threat model based on L ∞ perturbations, the convex set will be deﬁned as S 0 ( x ) =
Revised: For example, for a threat model based on L ∞ perturbations, the convex set will be deﬁned as S 0 ( x ) =

Original: { x (cid:48) ∈ R d 0 , || x − x (cid:48) || 
Revised: { x (cid:48) ∈ R d 0 , || x − x (cid:48) || ∞ < (cid:15) } .

Original: A neural network consisting of k layers and parameters θ is represented as a function h θ where h θ
Revised: A neural network consisting of k layers and parameters θ is represented as a function h θ where h θ

Original: = h kθ ◦ h k − 1 θ · · · ◦ h 1 θ and h iθ : R d i − 1 → R d i denotes a transformation applied at hidden layer i .
Revised: = h kθ ◦ h k − 1 θ · · · ◦ h 1 θ and h iθ : R d i − 1 → R d i denotes a transformation applied at hidden layer i .

Original: We also denote the function representing part of the neural network from layer i to the ﬁnal layer k as h i : kθ = h kθ ◦ h k − 1 θ · · · ◦ h iθ .
Revised: We also denote the function representing part of the neural network from layer i to the ﬁnal layer k as h i : kθ = h kθ ◦ h k − 1 θ · · · ◦ h iθ .

Original: Our goal will be to prove a property on the output of the neural network, encoded via a linear constraint:
Revised: Our goal will be to prove a property on the output of the neural network, encoded via a linear constraint:

Original: [Equation]
Revised: [Equation]

Original: 1) where c and d are property speciﬁc vector and scalar values.
Revised: 1) where c and d are property speciﬁc vector and scalar values.

Original: This formulation is general enough to capture many interesting safety properties (Dvijotham et al., 2018b; Qin et al., 2019), including robustness to L p perturbations.
Revised: This formulation is general enough to capture many interesting safety properties (Dvijotham et al., 2018b; Qin et al., 2019), including robustness to L p perturbations.

Original: The standard approach to train neural networks which satisfy this constraint is to deﬁne a surrogate loss L and solve the following min-max optimization problem:
Revised: The standard approach to train neural networks which satisfy this constraint is to deﬁne a surrogate loss L and solve the following min-max optimization problem:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Because inner maximization is intractable, most existing approaches replace it with an approximation.
Revised: Because inner maximization is intractable, most existing approaches replace it with an approximation.

Original: Depending on the used approximation, we distinguish two families of techniques.
Revised: Depending on the used approximation, we distinguish two families of techniques.

Original: Adversarial training One family of methods, referred to as adversarial training (Goodfellow et al., 2015; Kurakin et al., 2017), replaces the maximum loss with a lower bound which is obtained using an adversarial attack.
Revised: Adversarial training One family of methods, referred to as adversarial training (Goodfellow et al., 2015; Kurakin et al., 2017), replaces the maximum loss with a lower bound which is obtained using an adversarial attack.

Original: Madry et al.
Revised: Madry et al.

Original: (2018) maximized the inner loss using a projected gradient descent (PGD) attack and found that, perhaps surprisingly, this optimization procedure succeeds in training deep architectures which are robust against the strongest known adversaries.
Revised: (2018) maximized the inner loss using a projected gradient descent (PGD) attack and found that, perhaps surprisingly, this optimization procedure succeeds in training deep architectures which are robust against the strongest known adversaries.

Original: While this provides strong empirical evidence that the resulting models are indeed robust, the approach offers no guarantees.
Revised: While this provides strong empirical evidence that the resulting models are indeed robust, the approach offers no guarantees.

Original: Thus, it remains unclear whether there exist even stronger attacks that could break a model trained in this manner.
Revised: Thus, it remains unclear whether there exist even stronger attacks that could break a model trained in this manner.

Original: Provable defenses
Revised: Provable defenses

Original: A second family of methods to train certiﬁed neural networks is based on the computation of an upper bound to the inner loss, as opposed to a lower bound computed for adversarial training.
Revised: A second family of methods to train certiﬁed neural networks is based on the computation of an upper bound to the inner loss, as opposed to a lower bound computed for adversarial training.

Original: These methods are typically referred to as provable defenses as they provide guarantees on the robustness of the resulting network, under any kind of attack inside the threat model.
Revised: These methods are typically referred to as provable defenses as they provide guarantees on the robustness of the resulting network, under any kind of attack inside the threat model.

Original: An upper bound is typically computed using linear relaxations (Wong & Kolter, 2018), intervals (Gowal et al., 2018) or methods combining interval bounds and linear relaxations (Mirman et al., 2018;
Revised: An upper bound is typically computed using linear relaxations (Wong & Kolter, 2018), intervals (Gowal et al., 2018) or methods combining interval bounds and linear relaxations (Mirman et al., 2018;

Original: Zhang et al., 2019).
Revised: Zhang et al., 2019).

Original: However, these methods suffer from two disadvantages.
Revised: However, these methods suffer from two disadvantages.

Original: First, due to the convex relaxations, an upper bound on the loss is typically not tight and can be quite loose.
Revised: First, due to the convex relaxations, an upper bound on the loss is typically not tight and can be quite loose.

Original: However, we believe this is less of an issue due to the fact that interval relaxations were shown to experimentally be able to train more provably robust models than methods based on linear relaxations (which usually produce tighter bounds than intervals).
Revised: However, we believe this is less of an issue due to the fact that interval relaxations were shown to experimentally be able to train more provably robust models than methods based on linear relaxations (which usually produce tighter bounds than intervals).

Original: For example, Mirman et al.
Revised: For example, Mirman et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2019) report ∼ 28% robust error using pure interval training on CIFAR-10 with perturbation 8/255 while Wong et al.
Revised: (2019) report ∼ 28% robust accuracy using pure interval training on CIFAR-10 with perturbation 8/ while Wong et al.

Original: (2018) achieve 21% using linear relaxations.
Revised: (2018) achieve 21% using linear relaxations.

Original: Second, the way these methods construct the loss makes the relationship between the loss and the network parameters signiﬁcantly more complex than in standard training.
Revised: Second, the way these methods construct the loss makes the relationship between the loss and the network parameters signiﬁcantly more complex than in standard training.

Original: This causes the resulting optimization problem to be more difﬁcult, meaning these training methods often converge to a suboptimal solution.
Revised: This causes the resulting optimization problem to be more difﬁcult, meaning these training methods often converge to a suboptimal solution.

Original: Our experimental results conﬁrm this – we substantially outperform existing methods both in terms of accuracy and certiﬁed robustness using the same linear relaxation, but a different optimization procedure.
Revised: Our experimental results conﬁrm this – we substantially outperform existing methods both in terms of accuracy and certiﬁed robustness using the same linear relaxation, but a different optimization procedure.

Original: Certiﬁcation via convex relaxations We now formally describe how provable defenses perform certiﬁcation.
Revised: Certiﬁcation via convex relaxations We now formally describe how provable defenses perform certiﬁcation.

Original: We denote the set of possible intermediate concrete vectors at layer i that can be obtained by propagating vector x (cid:48) ∈ S 0 ( x ) through the network as S i ( x )
Revised: We denote the set of possible intermediate concrete vectors at layer i that can be obtained by propagating vector x (cid:48) ∈ S 0 ( x ) through the network as S i ( x )

Original: =
Revised: =

Original: h iθ ( S i − 1 ( x )) ⊆
Revised: h iθ ( S i − 1 ( x )) ⊆

Original: R d i .
Revised: R d i .

Original: As it is difﬁcult to explicitly compute the set S i ( x ) , a standard approach is to approximate it via a convex relaxation C i ( x ) .
Revised: As it is difﬁcult to explicitly compute the set S i ( x ) , a standard approach is to approximate it via a convex relaxation C i ( x ) .

Original: As the input set is already convex, there is no need to introduce a relaxation, and thus we set C 0 ( x )
Revised: As the input set is already convex, there is no need to introduce a relaxation, and thus we set C 0 ( x )

Original: =
Revised: =

Original: S 0 ( x ) .
Revised: S 0 ( x ) .

Original: Given a neural network layer h iθ which transforms one set of vectors into another, we represent its corresponding convex relaxation transformer as g i θ .
Revised: Given a neural network layer h iθ which transforms one set of vectors into another, we represent its corresponding convex relaxation transformer as g i θ .

Original: That is, g iθ will transform one convex set into another convex set.
Revised: That is, g iθ will transform one convex set into another convex set.

Original: More formally, for any set D ⊆ R d i − 1 , g iθ ( D ) is convex and h iθ ( D ) ⊆
Revised: More formally, for any set D ⊆ R d i − 1 , g iθ ( D ) is convex and h iθ ( D ) ⊆

Original: g iθ ( D ) .
Revised: g iθ ( D ) .

Original: Then, we recursively deﬁne the effect of g iθ on a convex relaxation as C i ( x ) =
Revised: Then, we recursively deﬁne the effect of g iθ on a convex relaxation as C i ( x ) =

Original: g iθ ( C i − 1 ( x )) ⊆
Revised: g iθ ( C i − 1 ( x )) ⊆

Original: R d i .
Revised: R d i .

Original: Finally, to certify robustness using the obtained convex relaxation, it is enough to check whether all output vectors in C k ( x ) satisfy the linear constraint in
Revised: Finally, to certify robustness using the obtained convex relaxation, it is enough to check whether all output vectors in C k ( x ) satisfy the linear constraint in

Original: Equation 1.
Revised: Equation 1.

Original: If this is true, then all output vectors in S k ( x ) satisfy the constraint as well due to the fact that S k ( x ) ⊆
Revised: If this is true, then all output vectors in S k ( x ) satisfy the constraint as well due to the fact that S k ( x ) ⊆

Original: C k ( x ) .
Revised: C k ( x ) .

Original: [Figure]
Revised: [Figure]

Original:  P ROVABLE D EFENSE VIA L AYERWISE ADVERSARIAL TRAINING
Revised:  P ROVABLE D EFENSE VIA L AYERWISE ADVERSARIAL TRAINING

Original: We now describe our layerwise adversarial training approach which yields a provable defense that bridges the gap between standard adversarial training and existing provable defenses.
Revised: We now describe our layerwise adversarial training approach which yields a provable defense that bridges the gap between standard adversarial training and existing provable defenses.

Original: Motivation: latent adversarial examples Consider an already trained neural network model h θ which we would like to certify using convex relaxations.
Revised: Motivation: latent adversarial examples Consider an already trained neural network model h θ which we would like to certify using convex relaxations.

Original: A fundamental issue here is that certiﬁcation methods based on convex relaxations can struggle to prove the target property (e.g., robustness) due to the iterative accumulation of errors introduced by the relaxation.
Revised: A fundamental issue here is that certiﬁcation methods based on convex relaxations can struggle to prove the target property (e.g., robustness) due to the iterative accumulation of errors introduced by the relaxation.

Original: More precisely, assume the neural network actually satisﬁes the property from Equation 1 for an input x , meaning that c T h θ ( x (cid:48) )
Revised: More precisely, assume the neural network actually satisﬁes the property from Equation 1 for an input x , meaning that c T h θ ( x (cid:48) )

Original: + d < 0 , ∀ x (cid:48) ∈ S 0 ( x ) .
Revised: + d < 0 , ∀ x (cid:48) ∈ S 0 ( x ) .

Original: Naturally, this also implies that the neural network behaves correctly in the latent space of its ﬁrst hidden layer in the region S 1 ( x ) .
Revised: Naturally, this also implies that the neural network behaves correctly in the latent space of its ﬁrst hidden layer in the region S 1 ( x ) .

Original: Formally, this means that c T h 2: k θ ( x (cid:48) 1 )+ d < 0 , ∀ x (cid:48) 1 ∈ S 1 ( x ) .
Revised: Formally, this means that c T h 2: k θ ( x (cid:48) 1 )+ d < 0 , ∀ x (cid:48) 1 ∈ S 1 ( x ) .

Original: However, if one would use a certiﬁcation method which replaces the region S 1 ( x ) by its convex relaxation C 1 ( x ) , then it is possible that we would fail to certify our desired property.
Revised: However, if one would use a certiﬁcation method which replaces the region S 1 ( x ) by its convex relaxation C 1 ( x ) , then it is possible that we would fail to certify our desired property.

Original: This is due to the fact that there may exist an input x (cid:48) 1 ∈ C 1 ( x ) \ S 1 ( x ) such that c T h 2: k θ ( x (cid:48) 1 ) + d ≥ 0 .
Revised: This is due to the fact that there may exist an input x (cid:48) 1 ∈ C 1 ( x ) \ S 1 ( x ) such that c T h 2: k θ ( x (cid:48) 1 ) + d ≥ 0 .

Original: Of course, we could repeat the above thought experiment and possibly ﬁnd more violating latent inputs in the set C i ( x ) \ S i ( x ) of any hidden layer i .
Revised: Of course, we could repeat the above thought experiment and possibly ﬁnd more violating latent inputs in the set C i ( x ) \ S i ( x ) of any hidden layer i .

Original: The existence of points found in the difference between a convex relaxation and the true region is a fundamental reason for the failure of certiﬁcation methods based on convex approximations.
Revised: The existence of points found in the difference between a convex relaxation and the true region is a fundamental reason for the failure of certiﬁcation methods based on convex approximations.

Original: For convenience, we refer to such points as latent adversarial examples .
Revised: For convenience, we refer to such points as latent adversarial examples .

Original: Next, we describe a method which trains the neural network in a way that aims to minimize the number of latent adversarial examples.
Revised: Next, we describe a method which trains the neural network in a way that aims to minimize the number of latent adversarial examples.

Original: Layerwise provable optimization via convex relaxations Our key observation is that the two families of defense methods described earlier are in fact different ends of the same spectrum: methods based on adversarial training maximize the cross-entropy loss in the ﬁrst convex region C 0 ( x ) while provable defenses maximize the same loss, but in the last convex region C k ( x ) .
Revised: Layerwise provable optimization via convex relaxations Our key observation is that the two families of defense methods described earlier are in fact different ends of the same spectrum: methods based on adversarial training maximize the cross-entropy loss in the ﬁrst convex region C 0 ( x ) while provable defenses maximize the same loss, but in the last convex region C k ( x ) .

Original: Both methods then backpropagate the loss through the network and update the parameters using SGD.
Revised: Both methods then backpropagate the loss through the network and update the parameters using SGD.

Original: However, as explained previously, certiﬁcation methods may fail even before the last layer due to the presence of latent adversarial examples in the difference of the regions C i ( x ) and S i ( x ) .
Revised: However, as explained previously, certiﬁcation methods may fail even before the last layer due to the presence of latent adversarial examples in the difference of the regions C i ( x ) and S i ( x ) .

Original: A natural question then is – can we leverage adversarial training so to eliminate latent adversarial examples from hidden layers and obtain a provable network?
Revised: A natural question then is – can we leverage adversarial training so to eliminate latent adversarial examples from hidden layers and obtain a provable network?

Original: To this end, we propose adversarial training in layerwise fashion.
Revised: To this end, we propose adversarial training in layerwise fashion.

Original: The initial phase of training is equivalent to adversarial training as used by Madry et al.
Revised: The initial phase of training is equivalent to adversarial training as used by Madry et al.

Original: In this phase in the inner loop we repeatedly ﬁnd an input in C 0 ( x ) which maximizes the cross-entropy loss and update the parameters of the neural network so to minimize this loss using SGD.
Revised: In this phase in the inner loop we repeatedly ﬁnd an input in C 0 ( x ) which maximizes the cross-entropy loss and update the parameters of the neural network so to minimize this loss using SGD.

Original: Note that the outcome of this phase is a model which is highly robust against strong multi-step adversaries.
Revised: Note that the outcome of this phase is a model which is highly robust against strong multi-step adversaries.

Original: However, certiﬁcation of this fact often fails due to the previously mentioned accumulation of errors in the particular convex relaxation being used.
Revised: However, certiﬁcation of this fact often fails due to the previously mentioned accumulation of errors in the particular convex relaxation being used.

Original: The next step of our training method is visually illustrated in Figure 1.
Revised: The next step of our training method is visually illustrated in Figure 1.

Original: Here, we propagate the initial convex region through the ﬁrst layer of the network and obtain the convex relaxation C 1 ( x ) .
Revised: Here, we propagate the initial convex region through the ﬁrst layer of the network and obtain the convex relaxation C 1 ( x ) .

Original: We then solve the optimization problem to ﬁnd a concrete point x (cid:48)
Revised: We then solve the optimization problem to ﬁnd a concrete point x (cid:48)

Original: 1 inside of C 1 ( x ) which produces
Revised: 1 inside of C 1 ( x ) which produces

Original: Data: k -layer neural network h
Revised: Data: k -layer neural network h

Original: θ , training set ( X , Y ) , learning rate η , step size α , inner steps n
Revised: θ , training set ( X , Y ) , learning rate η , step size α , inner steps n

Original: Result:
Revised: Result:

Original: Certiﬁably robust neural network h θ 1 for l ≤ k do for i ≤ n epochs do
Revised: Certiﬁably robust neural network h θ 1 for l ≤ k do for i ≤ n epochs do

Original: Sample mini-batch { ( x 1 , y 1 ) , ( x 2 , y 2 ) , ..., ( x b , y b ) } ∼ ( X , Y ) ;
Revised: Sample mini-batch { ( x 1 , y 1 ) , ( x 2 , y 2 ) , ..., ( x b , y b ) } ∼ ( X , Y ) ;

Original: Compute convex relaxations C l ( x 1 ) , C l ( x 2 ) , ..., C l ( x k ) ;
Revised: Compute convex relaxations C l ( x 1 ) , C l ( x 2 ) , ..., C l ( x b ) ;

Original: Initialize x (cid:48) 1 ∼ C l ( x 1 ) , x (cid:48)
Revised: Initialize x (cid:48) 1 ∼ C l ( x 1 ) , x (cid:48)

Original: 2 ∼ C l ( x 2 ) , ..., x (cid:48)
Revised: 2 ∼ C l ( x 2 ) , ..., x (cid:48)

Original: b ∼ C l ( x b ) ; for j ≤ b do
Revised: b ∼ C l ( x b ) ; for j ≤ b do

Original: Update in parallel  x (cid:48)
Revised: Update in parallel n times: x (cid:48)

Original: j ← Π C l ( x j ) ( x (cid:48)
Revised: j ← Π C l ( x j ) ( x (cid:48)

Original: j
Revised: j

Original: + α ∇ x (cid:48)
Revised: + α ∇ x (cid:48)

Original: j L ( h j +1:
Revised: j L ( h j +1:

Original: k θ ( x (cid:48) j ) , y j )) ; end
Revised: k θ ( x (cid:48) j ) , y j )) ; end

Original: Update parameters θ ← θ − η 1 M (cid:80) bj =1 ∇ θ L ( h j +1:
Revised: Update parameters θ ← θ − η 1 M (cid:80) bj =1 ∇ θ L ( h j +1:

Original: k θ ( x (cid:48) j ) , y j ) ; end
Revised: k θ ( x (cid:48) j ) , y j ) ; end

Original: Freeze parameters θ l  of layer l  ; 12 end the maximum loss when this point is propagated further through the network (this forward pass is shown with the blue line).
Revised: Freeze parameters θ l +1 of layer l + 1 ; 12 end the maximum loss when this point is propagated further through the network (this forward pass is shown with the blue line).

Original: Finally, we backpropagate the ﬁnal loss (red line) and update the parameters of the network so to minimize the loss.
Revised: Finally, we backpropagate the ﬁnal loss (red line) and update the parameters of the network so to minimize the loss.

Original: Critically, we do not backpropagate through the convex relaxation in the ﬁrst layer as standard provable defenses do (Wong & Kolter, 2018; Mirman et al., 2018; Gowal et al., 2018).
Revised: Critically, we do not backpropagate through the convex relaxation in the ﬁrst layer as standard provable defenses do (Wong & Kolter, 2018; Mirman et al., 2018; Gowal et al., 2018).

Original: We instead freeze the ﬁrst layer and stop backpropagation after the update of the second layer.
Revised: We instead freeze the ﬁrst layer and stop backpropagation after the update of the second layer.

Original: Because of this, our optimization problem is signiﬁcantly easier – the neural network only has to learn to behave well on the concrete points that were found in the convex region C l ( x ) .
Revised: Because of this, our optimization problem is signiﬁcantly easier – the neural network only has to learn to behave well on the concrete points that were found in the convex region C l ( x ) .

Original: This can be viewed as an extension of the robust optimization method that Madry et al.
Revised: This can be viewed as an extension of the robust optimization method that Madry et al.

Original: (2018) found to work well in practice.
Revised: (2018) found to work well in practice.

Original: We then proceed with the above process for later layers.
Revised: We then proceed with the above process for later layers.

Original: Formally, this training process amounts to (approximately) solving the following min-max optimization problem at the l -th step:
Revised: Formally, this training process amounts to (approximately) solving the following min-max optimization problem at the l -th step:

Original: [Equation]
Revised: [Equation]

Original: Note that for l = 0 this formulation is equivalent to the standard min-max formulation in Equationbecause C 0 ( x ) =
Revised: Note that for l = 0 this formulation is equivalent to the standard min-max formulation in Equationbecause C 0 ( x ) =

Original: S 0 ( x ) .
Revised: S 0 ( x ) .

Original: Our approach to solve this min-max optimization problem for every layer l is shown in Algorithm 1.
Revised: Our approach to solve this min-max optimization problem for every layer l is shown in Algorithm 1.

Original: We initialize every batch by random sampling from the corresponding convex region.
Revised: We initialize every batch by random sampling from the corresponding convex region.

Original: Then, in every iteration we use projected gradient descent (PGD) to maximize the inner loss in 3.
Revised: Then, in every iteration we use projected gradient descent (PGD) to maximize the inner loss in 3.

Original: We ﬁrst update x (cid:48)
Revised: We ﬁrst update x (cid:48)

Original: j in the direction of the gradient of the loss
Revised: j in the direction of the gradient of the loss

Original: and then project it back to C l ( x j ) using the projection operator Π .
Revised: and then project it back to C l ( x j ) using the projection operator Π .

Original: Note that this approach assumes the existence of an efﬁcient projection method to the particular convex relaxation the method is instantiated with.
Revised: Note that this approach assumes the existence of an efﬁcient projection method to the particular convex relaxation the method is instantiated with.

Original: In the next section, we show how to instantiate the training algorithm described above to a particular convex relaxation which is generally tighter than a hyperrectangle and where we derive an efﬁcient projection operation.
Revised: In the next section, we show how to instantiate the training algorithm described above to a particular convex relaxation which is generally tighter than a hyperrectangle and where we derive an efﬁcient projection operation.

Original:  L AYERWISE ADVERSARIAL TRAINING USING LINEAR RELAXATIONS
Revised:  L AYERWISE ADVERSARIAL TRAINING USING LINEAR RELAXATIONS

Original: So far we described the general approach of layerwise adversarial training.
Revised: So far we described the general approach of layerwise adversarial training.

Original: Now we show how to instantiate it for a particular convex relaxation based on linear approximations.
Revised: Now we show how to instantiate it for a particular convex relaxation based on linear approximations.

Original: If instead one would use interval approximation (Mirman et al., 2018; Gowal et al., 2018) as the convex relaxation, then all regions C l ( x ) will be hyperrectangles and projection to these sets is fast and simple.
Revised: If instead one would use interval approximation (Mirman et al., 2018; Gowal et al., 2018) as the convex relaxation, then all regions C l ( x ) will be hyperrectangles and projection to these sets is fast and simple.

Original: However, the interval relaxation provides a coarse approximation which motivates the need to train with relaxations that provide tighter bounds.
Revised: However, the interval relaxation provides a coarse approximation which motivates the need to train with relaxations that provide tighter bounds.

Original: Thus, we consider linear relaxations which are generally tighter than those based on intervals.
Revised: Thus, we consider linear relaxations which are generally tighter than those based on intervals.

Original: In particular we leverage the same relaxation which was previously proposed in Wong & Kolter (2018); Weng et al.
Revised: In particular we leverage the same relaxation which was previously proposed in Wong & Kolter (2018); Weng et al.

Original: ; Singh et al.
Revised: ; Singh et al.

Original: as an effective way to certify neural networks.
Revised: as an effective way to certify neural networks.

Original: Here, each convex region is represented as a set C l ( x ) =
Revised: Here, each convex region is represented as a set C l ( x ) =

Original: { a l + A l e | e ∈ [ − 1 , 1] m l } .
Revised: { a l + A l e | e ∈ [ − 1 , 1] m l } .

Original: Vector a l represents the center of the set and the matrix A l represents the afﬁne transformation of the hypercube eee Π( e )
Revised: Vector a l represents the center of the set and the matrix A l represents the afﬁne transformation of the hypercube eee Π( e )

Original: C l ( x ) x (cid:48)x (cid:48)x (cid:48) Π( x (cid:48) )
Revised: C l ( x ) x (cid:48)x (cid:48)x (cid:48) Π( x (cid:48) )

Original: x (cid:48) 1 := 2 e 1 − ex (cid:48) 
Revised: x (cid:48) 1 := 2 e 1 − ex (cid:48) 2 :=

Original: [Figure] := [Figure]
Revised: e 1 + e

Original: [ − 1 , 1] m l .
Revised: [ − 1 , 1] m l .

Original: The initial convex region C 0 ( x ) is represented using a 0 = x and A 

Revised: The initial convex region C 0 ( x ) is represented using a 0 = x and A

Original: = (cid:15) I d 0 is a diagonal matrix.
Revised: = (cid:15) I d 0 is a diagonal matrix.

Original: Propagation of these convex regions through the network is out of the scope of this paper – a full description can be found in Wong & Kolter (2018) or Singh et al.
Revised: Propagation of these convex regions through the network is out of the scope of this paper – a full description can be found in Wong & Kolter (2018) or Singh et al.

Original: At a high level, the convolutional and fully connected layers are handled by multiplying A l and a l by appropriate matrices.
Revised: At a high level, the convolutional and fully connected layers are handled by multiplying A l and a l by appropriate matrices.

Original: To handle the ReLU activation, for ReLU units which cross 0, we apply a convex relaxation which amounts to multiplying A l and a l by appropriately chosen scalar values, depending whether the ReLU is activated or not.
Revised: To handle the ReLU activation, for ReLU units which cross 0, we apply a convex relaxation which amounts to multiplying A l and a l by appropriately chosen scalar values, depending whether the ReLU is activated or not.

Original: Using this relaxation of ReLU, we can recursively obtain all convex regions C l ( x ) .
Revised: Using this relaxation of ReLU, we can recursively obtain all convex regions C l ( x ) .

Revised: In practice, A l e can be computed without explicitly constructing matrix A l because

Revised: A l e = W l Λ l − 1 W l − 2 · · · M 0 e .

Revised: Then we can perform matrix-vector multiplication right to left to obtain vector A l e .

Original: We provide more detailed description of this propagation in Appendix A.1.
Revised: We provide more detailed description of this propagation in Appendix A.

Original: Projection to linear convex regions To use our training method we now need to instantiate Algorithm 1 with a suitable projection operator Π C l ( x ) .
Revised: Projection to linear convex regions To use our training method we now need to instantiate Algorithm 1 with a suitable projection operator Π C l ( x ) .

Original: The key insight here is that the vector x
Revised: The key insight here is that the vector x

Original: (cid:48) ∈
Revised: (cid:48) ∈

Original: C l ( x ) is uniquely determined by auxiliary vector e ∈ [ − 1 , 1] m l where x (cid:48)
Revised: C l ( x ) is uniquely determined by auxiliary vector e ∈ [ − 1 , 1] m l where x (cid:48)

Original: = a l + A l e .
Revised: = a l + A l e .

Original: Then instead of directly solving for x (cid:48) which requires projecting to C l ( x ) , we can solve for e instead which would uniquely determine x .
Revised: Then instead of directly solving for x (cid:48) which requires projecting to C l ( x ) , we can solve for e instead which would uniquely determine x .

Original: Crucially, the domain of e is a hyperrectangle [ − 1 , 1] m l which is easy to project to.
Revised: Crucially, the domain of e is a hyperrectangle [ − 1 , 1] m l which is easy to project to.

Original: To visualize this further we provide an example in Figure 2.
Revised: To visualize this further we provide an example in Figure 2.

Original: The goal is to project the red point x (cid:48) in the right picture to the we ﬁrst perform then
Revised: The goal is to project the red point x (cid:48) in the right picture to the convex region C l ( x ) .

Original: Then again 
Revised: To project, we ﬁrst perform change of variables to substitute x (cid:48) with e and then project e to the square [ − 1 , 1] × [ − 1 , 1] to obtain the blue point Π( e ) on the left. Then, we again perform change of variables to obtain the blue point Π ( x (cid:48) ) on the right, the projection of x (cid:48)

Original:  Based on these observations, we modify Line 7 of Algorithm 1 to ﬁrst update the coefﬁcients e j using the following update rule: e j ← clip ( e j + α A Tl ∇ x (cid:48)
Revised: we were looking for. Based on these observations, we modify Line 7 of Algorithm 1 to ﬁrst update the coefﬁcients e j using the following update rule: e j ← clip ( e j + α A Tl ∇ x (cid:48)

Original: j L ( x (cid:48)
Revised: j L ( x (cid:48)

Original: j , y j ) , − 1 , 1) .
Revised: j , y j ) , − 1 , 1) .

Original: Here clip is function which thresholds its argument between -1 and 1, formally clip ( x, − 1 , 1)
Revised: Here clip is function which thresholds its argument between -1 and 1, formally clip ( x, − 1 , 1)

Original: = min(max( x, − 1) , 1) .
Revised: = min(max( x, − 1) , 1) .

Original: This is followed by an update to x (cid:48)
Revised: This is followed by an update to x (cid:48)

Original: j via x (cid:48)
Revised: j via x (cid:48)

Original: j ← a l + A l e j , completing the update step.
Revised: j ← a l + A l e j , completing the update step.

Original: Sparse representation While our representation of convex regions with matrix A l and vector a l has clean mathematical properties, in practice, a possible issue is that the matrix A l can grow to be quite large.
Revised: Sparse representation While our representation of convex regions with matrix A l and vector a l has clean mathematical properties, in practice, a possible issue is that the matrix A l can grow to be quite large.

Original: Because of this, propagating it through the network can be memory intensive and prohibit the use of larger batches.
Revised: Because of this, propagating it through the network can be memory intensive and prohibit the use of larger batches.

Original: To overcome this difﬁculty, we ﬁrst observe that A l is quite sparse.
Revised: To overcome this difﬁculty, we ﬁrst observe that A l is quite sparse.

Original: We start with a very sparse, diagonal matrix
Revised: We start with a very sparse, diagonal matrix

Original: A 0 at the input.
Revised: A 0 at the input.

Original: After each convolution, an element of matrix A l +1 is non-zero only if there is a non-zero element inside of its convolutional kernel in matrix A l .
Revised: After each convolution, an element of matrix A l +1 is non-zero only if there is a non-zero element inside of its convolutional kernel in matrix A l .

Original: We can leverage this observation to precompute positions of all non-zero elements in matrix
Revised: We can leverage this observation to precompute positions of all non-zero elements in matrix

Original: A l +1 and compute their values using matrix multiplication.
Revised: A l +1 and compute their values using matrix multiplication.

Original: This optimization is critical to enabling training to take place altogether.
Revised: This optimization is critical to enabling training to take place altogether.

Original: An interesting item for future work is further optimizing the current relaxation (via a specialized GPU implementation) or developing more memory friendly relaxations, so to scale the training to larger networks.
Revised: An interesting item for future work is further optimizing the current relaxation (via a specialized GPU implementation) or developing more memory friendly relaxations, so to scale the training to larger networks.

Original:  C ERTIFICATION OF NEURAL NETWORKS
Revised:  C ERTIFICATION OF NEURAL NETWORKS

Original: After training a neural network via layerwise adversarial training, our goal is to certify the target property (e.g., robustness).
Revised: After training a neural network via layerwise adversarial training, our goal is to certify the target property (e.g., robustness).

Original: Here we leverage certiﬁcation techniques which are not fast enough to be incorporated into the training procedure, but which can signiﬁcantly boost the certiﬁcation performance.
Revised: Here we leverage certiﬁcation techniques which are not fast enough to be incorporated into the training procedure, but which can signiﬁcantly boost the certiﬁcation performance.

Original: Reﬁnement of the linear approximation The linear relaxation of ReLU that we are using is parameterized by slopes λ of the linear relaxation.
Revised: Reﬁnement of the linear approximation The linear relaxation of ReLU that we are using is parameterized by slopes λ of the linear relaxation.

Original: Prior work which employed this relaxation (Wong & Kolter, 2018; Weng et al., 2018; Singh et al., 2018) chose these slopes in a greedy manner by minimizing the area of the relaxation.
Revised: Prior work which employed this relaxation (Wong & Kolter, 2018; Weng et al., 2018; Singh et al., 2018) chose these slopes in a greedy manner by minimizing the area of the relaxation.

Original: During training we also choose λ in the same way.
Revised: During training we also choose λ in the same way.

Original: However, during certiﬁcation, we can also optimize for the values of λ that give rise to the convex region inside of which the maximum loss is minimized.
Revised: However, during certiﬁcation, we can also optimize for the values of λ that give rise to the convex region inside of which the maximum loss is minimized.

Original: This optimization problem can be written as:
Revised: This optimization problem can be written as:

Original: [Equation]
Revised: [Equation]

Original: Solving this is computationally too expensive inside the training loop, but during certiﬁcation it is feasible to approximate the solution.
Revised: Solving this is computationally too expensive inside the training loop, but during certiﬁcation it is feasible to approximate the solution.

Original: We solve for λ using the Adam optimizer and clipping the elements between 0 and 1 after each update.
Revised: We solve for λ using the Adam optimizer and clipping the elements between 0 and 1 after each update.

Original: We remark that the idea of learning the slope is similar to Dvijotham et al.
Revised: We remark that the idea of learning the slope is similar to Dvijotham et al.

Original: (2018b) who propose to optimize dual variables in a dual formulation, however here we stay in the primal formulation.
Revised: (2018b) who propose to optimize dual variables in a dual formulation, however here we stay in the primal formulation.

Original: Combining convex relaxations with exact bound propagation During layerwise adversarial training we essentially train the network to be certiﬁed on all regions C 0 ( x ) , ..., C k ( x ) .
Revised: Combining convex relaxations with exact bound propagation During layerwise adversarial training we essentially train the network to be certiﬁed on all regions C 0 ( x ) , ..., C k ( x ) .

Original: While computing exact regions S l ( x ) ⊆
Revised: While computing exact regions S l ( x ) ⊆

Original: C l ( x ) is not feasible during training, we can afford it to some extent during certiﬁcation.
Revised: C l ( x ) is not feasible during training, we can afford it to some extent during certiﬁcation.

Original: The idea is to ﬁrst propagate the bounds using convex relaxations until one of the hidden layers l and obtain a region C l ( x ) .
Revised: The idea is to ﬁrst propagate the bounds using convex relaxations until one of the hidden layers l and obtain a region C l ( x ) .

Original: If training was successful, there should not exist a concrete point x (cid:48) l ∈ C l ( x ) which, if propagated through the network, violates the correctness property in Equation 1.
Revised: If training was successful, there should not exist a concrete point x (cid:48) l ∈ C l ( x ) which, if propagated through the network, violates the correctness property in Equation 1.

Original: We can encode both, the property and the propagation of the exact bounds S l ( x ) using a Mixed-Integer Linear Programming (MILP) solver.
Revised: We can encode both, the property and the propagation of the exact bounds S l ( x ) using a Mixed-Integer Linear Programming (MILP) solver.

Original: Note that we can achieve this because we represent the region C l ( x ) using a set of linear constraints, however, for general convex shapes this may not be possible.
Revised: Note that we can achieve this because we represent the region C l ( x ) using a set of linear constraints, however, for general convex shapes this may not be possible.

Original: We perform the MILP encoding using the formulation from Tjeng et al.
Revised: We perform the MILP encoding using the formulation from Tjeng et al.

Original: It is usually possible to encode only the last two layers using MILP due to the poor scalability of these solvers for realistic network sizes.
Revised: It is usually possible to encode only the last two layers using MILP due to the poor scalability of these solvers for realistic network sizes.

Original: One further improvement we also include is to tighten the convex regions C l ( x ) using reﬁnement via linear programming as described in Singh et al. (2019a).
Revised: One further improvement we also include is to tighten the convex regions C l ( x ) using reﬁnement via linear programming as described in Singh et al. (2019a).

Original: We remark that this combination of convex relaxation and exact bound propagation does not fall under the recently introduced convex barrier to certiﬁcation Salman et al.
Revised: We remark that this combination of convex relaxation and exact bound propagation does not fall under the recently introduced convex barrier to certiﬁcation Salman et al.

Original: (2019b).
Revised: (2019b).

Original:  E XPERIMENTAL E VALUATION
Revised:  E XPERIMENTAL E VALUATION

Original: We now present an evaluation of our training method on the challenging CIFAR-10 dataset.
Revised: We now present an evaluation of our training method on the challenging CIFAR-10 dataset.

Original: Experimental setup We evaluate on a desktop PC with 2 GeForce RTX 2080 Ti GPU-s and 16core Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz.
Revised: Experimental setup We evaluate on a desktop PC with 2 GeForce RTX 2080 Ti GPU-s and 16core Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz.

Original: We use Gurobi as a MILP solver.
Revised: We use Gurobi as a MILP solver.

Original: Our method is implemented in PyTorch and we plan to release both, the code and the trained models.
Revised: Our method is implemented in PyTorch and we plan to release both, the code and the trained models.

Original: Neural network architecture All presented results are on a 3-layer convolutional network with 16 544 neurons: ﬁrst 2 layers are convolutional layers with  32 and 128 ﬁlters, and both have kernel size 4 and stride 2.
Revised: Neural network architecture All presented results are on a 4-layer convolutional network with 49 402 neurons: ﬁrst 3 layers are convolutional layers with ﬁlter sizes 32, 32 and 128, kernel sizes 3, 4, 4 and strides 1, 2, 2 respectively.

Original: These are followed by a fully connected layer with 150 hidden units.
Revised: These are followed by a fully connected layer with 250 hidden units.

Original: After each of these layers, there is a ReLU activation.
Revised: After each of these layers, there is a ReLU activation.

Original: Training We use batch size 50 and L1 regularization 0.00001 for training.
Revised: Training We use batch size 50 and L1 regularization 0.00001 for training.

Original: We perform optimization using Adam (Kingma & Ba, 2014) with initial learning rate 0.001 which is decreased by 10 × every 100 epochs.
Revised: We perform optimization using Adam (Kingma & Ba, 2014) with initial learning rate 0.001 which is decreased by 10 × every 100 epochs.

Original: During layerwise training we start with (cid:15) perturbation which is 10% higher than the one we certify and we decrease it by 5% when the training progresses to the next layer.
Revised: During layerwise training we start with (cid:15) perturbation which is 10% higher than the one we certify and we decrease it by 5% when the training progresses to the next layer.

Original: Certiﬁcation After training completes, we perform certiﬁcation as follows: for every image, we ﬁrst try to certify it using only linear relaxations (with the improvement of learned slopes, Section 6).
Revised: Certiﬁcation After training completes, we perform certiﬁcation as follows: for every image, we ﬁrst try to certify it using only linear relaxations (with the improvement of learned slopes, Section 6).

Original: If this fails, we encode the last layer as MILP and try again.
Revised: If this fails, we encode the last layer as MILP and try again.

Original: Finally, if this fails we encode the
Revised: Finally, if this fails we encode the

Original: ReLU activation after the last convolution using additional 300 binary variables and the rest using the triangle formulation Ehlers (2017).
Revised: ReLU activation after the last convolution using additional 300 binary variables and the rest using the triangle formulation Ehlers (2017).

Original: We consider an image to be not certiﬁable if we fail to verify it using these methods.
Revised: We consider an image to be not certiﬁable if we fail to verify it using these methods.

Original: We always evaluate on the ﬁrst 1 000 images from the test set.
Revised: We always evaluate on the ﬁrst 1 000 images from the test set.

Original: Comparison to prior work We ﬁrst train and certify using our method for the L ∞ perturbation 2/255.
Revised: Comparison to prior work We ﬁrst train and certify using our method for the L ∞ perturbation 2/255.

Original: Results are shown in Table 1.
Revised: Results are shown in Table 1.

Original: We always compare to the best reported and reproducible
Revised: We always compare to the best reported and reproducible

Original: [Table] [Table]
Revised: [Table] [Table]

Original: results in the literature on any architecture.
Revised: results in the literature on any architecture.

Original: We do not compare to smoothing-based approaches
Revised: We do not compare to smoothing-based approaches

Original: Cohen et al.
Revised: Cohen et al.

Original: (2019), as these provide probabilistic instead of exact guarantees.
Revised: (2019), as these provide probabilistic instead of exact guarantees.

Original: Extensions to Cohen et al.
Revised: Extensions to Cohen et al.

Original: (2019) such as Salman et al.
Revised: (2019) such as Salman et al.

Original: (2019a) also use additional existing techniques such as pre-training on ImageNet and unlabeled data which are orthogonal.
Revised: (2019a) also use additional existing techniques such as pre-training on ImageNet and unlabeled data which are orthogonal.

Original: We also do not compare to using cascades from Wong et al.
Revised: We also do not compare to using cascades from Wong et al.

Original: (2018), as this improvement is also orthogonal to the method here.
Revised: (2018), as this improvement is also orthogonal to the method here.

Original: Thus, we only consider their best single network architecture (inline with prior work Zhang et al.
Revised: Thus, we only consider their best single network architecture (inline with prior work Zhang et al.

Original: (2019) which compares to a single architecture).
Revised: (2019) which compares to a single architecture).

Original: We believe all methods listed in Table 1, including ours, would beneﬁt from additional techniques such as cascades, pre-training and leveraging unlabeled data.
Revised: We believe all methods listed in Table 1, including ours, would beneﬁt from additional techniques such as cascades, pre-training and leveraging unlabeled data.

Original: Experimentally, we ﬁnd that the neural network trained using our method substantially outperforms all existing approaches, both in terms of standard accuracy and certiﬁed robustness for 2/255.
Revised: Experimentally, we ﬁnd that the neural network trained using our method substantially outperforms all existing approaches, both in terms of standard accuracy and certiﬁed robustness for 2/255.

Original: Note that here we are using the same linear relaxation as Wong et al.
Revised: Note that here we are using the same linear relaxation as Wong et al.

Original: [Table]

Original:  but our optimization procedure is different and shows signiﬁcant improvements over the one used in their work.
Revised: , but our optimization procedure is different and shows signiﬁcant improvements over the one used in their work.

Original: We also run the same experiment for L ∞ perturbation 8/255.
Revised: We also run the same experiment for L ∞ perturbation 8/255.

Original: Here we do not include comparison with Gowal et al.
Revised: Here we do not include comparison with Gowal et al.

Original: (2018) as their results were found to be not reproducible (Zhang et al., 2019; Mirman et al., 2019; Xu, 2019).
Revised: (2018) as their results were found to be not reproducible (Zhang et al., 2019; Mirman et al., 2019; Xu, 2019).

Original: These results are presented in Table 2.
Revised: These results are presented in Table 2.

Original: Here wematch state-of-the-art accuracy 46.2% reported by Mirman et al. (2018) and substantially outperform all other approaches.
Revised: Here we substantially outperform all existing approaches in terms of standard accuracy.

Original: However, in terms of certiﬁed robustness we are not able to achieve similar results to Zhang et al.
Revised: However, in terms of certiﬁed robustness we are not able to achieve similar results to Zhang et al.

Original: (2019) whose method is based on a combination of interval approximation and linear relaxation.
Revised: (2019) whose method is based on a combination of interval approximation and linear relaxation.

Original: The main issue is that our 3-layer network lacks capacity to solve this task – even if training only using standard adversarial training our empirical robustness does not go above ∼ 32% .
Revised: The main issue is that our 4-layer network lacks capacity to solve this task – even if training only using standard adversarial training our empirical robustness does not go above ∼ 34% .

Original: We remark that capacity was found to be one of the key components necessary to obtain a robust classiﬁer (Madry et al., 2018).
Revised: We remark that capacity was found to be one of the key components necessary to obtain a robust classiﬁer (Madry et al., 2018).

Original: Indeed, prior work achieves best results using signiﬁcantly larger networks, e.g. Mirman et al.

Original: (2018) achieve these results on a network with ∼ 170000 neurons, more than 10 times larger than ours.

Original: Due to promising results for 2/255, we believe achieving state-of-the-art results for 8/ is very likely an issue of instantiating our method with a convex relaxation that is more memory efﬁcient, which we believe is an interesting item for future work.
Revised: Due to promising results for 2/255, we believe achieving state-of-the-art results for 8/255 is very likely an issue of instantiating our method with a convex relaxation that is more memory efﬁcient, which we believe is an interesting item for future work.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We presented a new method to train certiﬁed neural networks.
Revised: We presented a new method to train certiﬁed neural networks.

Original: The key concept was to combine techniques from provable defenses using convex relaxations with those of adversarial training.
Revised: The key concept was to combine techniques from provable defenses using convex relaxations with those of adversarial training.

Original: Our method achieves state-of-the-art 74.8% accuracy and 55.9% certiﬁed robustness on CIFAR-10 with a 2/255 L ∞ perturbation, signiﬁcantly outperforming prior work when considering a single network (it also achieves competitive results on 8/255 L ∞ ).
Revised: Our method achieves state-of-the-art 78.8% accuracy and 58.1% certiﬁed robustness on CIFAR-10 with a 2/255 L ∞ perturbation, signiﬁcantly outperforming prior work when considering a single network (it also achieves competitive results on 8/255 L ∞ ).

Original: The method is general and can be instantiated with any convex relaxation.
Revised: The method is general and can be instantiated with any convex relaxation.

Original: A promising future work item is scaling to larger networks: this will require tight convex relaxations with a low memory footprint that allow for efﬁcient projection.
Revised: A promising future work item is scaling to larger networks: this will require tight convex relaxations with a low memory footprint that allow for efﬁcient projection.

Original: A A PPENDIX

Original: Here we provide additional details that were omitted in the main body of the paper.

Original: A. C ONVEX RELAXATION USING LINEAR APPROXIMATIONS
Revised: A C ONVEX RELAXATION USING LINEAR APPROXIMATIONS

Original: Here we describe how we propagate convex relaxations of the form C l ( x ) = { a l + A l e | e ∈ [ − 1 , 1] m l } through the network.
Revised: In this section, we describe how to propagate convex relaxations of the form C l ( x ) = { a l + A l e | e ∈ [ − 1 , 1] m l } through the network, for a single input x .

Original: As explained before, these relaxations were previously proposed in Wong & Kolter (2018); Weng et al.
Revised: As explained before, these relaxations were previously proposed in Wong & Kolter (2018); Weng et al.

Original: ; Singh et al.
Revised: ; Singh et al.

Original: For the sake of completeness we describe them here using our notation.
Revised: For the sake of completeness we describe them here using our notation.

Original: [-The initial convex region C 0 ( x ) is represented using a 0 = x and A 
-]
Revised: [-The initial convex region C 0 ( x ) is represented using a 0 = x and A 
-]

Original: = (cid:15) I d 0 is a diagonal matrix.

Original: Now we describe how to propagate these regions through different operations in the neural network.

Original: Depending on the form of function h iθ representing operation applied at layer i we distinguish different cases.
Revised: Depending on the form of function h iθ representing operation applied at layer i we distinguish different cases.

Original: Here we assume we have obtained region C i  ( x ) and our goal is to compute the region C i+1 ( x ) using convex relaxation g iθ of the function h iθ .
Revised: Here we assume we have obtained region C i − 1 ( x ) and our goal is to compute the region C i ( x ) using convex relaxation g iθ of the function h iθ .

Original: Convolutional layers Consider the case where h iθ is a convolution parametrized by kernel matrix W θ and bias b θ meaning that h iθ ( x (cid:48) ) =
Revised: Initial convex region Let (cid:15) be L ∞ radius that we are certifying.

Original: conv ( x  , W θ ) + b θ . Recall that x (cid:48) = a l + A l e where e ∈ [ − 1 , 1]
Revised: We then compute minimum and maximum pixel values for each pixel as x l = max(0 , x − (cid:15) ) and x u = min(1 , x + (cid:15) ) .

Original: m l . We can then compute:
Revised: We deﬁne initial convex region as:

Original: [Equation]
Revised: [Equation]

Original: Using this formula, we can deﬁne region C l +1 ( x )
Revised: Convolutional and fully connected layers For both convolutional and fully connected layers, the update is given by x (cid:48)

Original: {+i +
+} = { a l +1 + A l +1 e | e ∈ [ − 1 , 1] m l +1 } where:
Revised: {+i +
+} = h i +1 θ

Original: [Equation] Fully-connected layers Consider the case where h iθ is a fully connected layer parametrized by weight matrix W θ and bias b θ meaning that h iθ ( x (cid:48)  )
Revised: ( x (cid:48) i )

Original: =
Revised: =

Original: W θ x (cid:48)
Revised: W i +1 x (cid:48)

Original:  + b θ .
Revised: i + b i +1 .

Original: Recall that x (cid:48)
Revised: Recall that x (cid:48)

Original:  = a l + A l e where e ∈ [ − 1 , 1] m l .
Revised: i = a i + A i e where e ∈ [ − 1 , 1] m i .

Original: We can then compute:
Revised: We can then compute:

Original: [Equation]
Revised: [Equation]

Original: Using this formula, wecan deﬁne  region C l +1 ( x ) 
Revised: Using this formula, we deﬁne convex region C i +1 ( x ) =

Original: = { a l +1 + A l +1 e | e ∈ [ − 1 , 1] m l +1 } where: 
Revised: { a i +1 + A i +1 e | e ∈ [ − 1 , 1] m i +1 } where: m i +1 : = m i a i +1 : = W i +1 a i

Revised: + b i +

Revised: A i +1 :

Original: [Equation]{+= W i +
+}
Revised: {+= W i +
+}

Original:  ReLU activation In this casewe can assume that h iθ is ReLU function, h iθ ( x (cid:48))
Revised: A i ReLU activation In this case h i +1 θ is ReLU activation function x (cid:48)

Original: {+i +
+}
Revised: {+i +
+}

Revised: = h

Revised: i +1 θ

Revised: ( x (cid:48) i ) =

Revised: ReLU ( x (cid:48) i )

Original: =
Revised: =

Original: max( x (cid:48)  , 0) applied componentwise.
Revised: max( x (cid:48) i , 0) applied componentwise.

Original: We will explain the transformation of a single element x (cid:48)
Revised: We will explain the transformation of a single element x (cid:48)

Original: i = a l + A l,i e .
Revised: i,j = a i,j + A Ti,j e .

Original: We ﬁrst compute lower  and upper boundl i , u i that this element can take in the convex region C l ( x ) :l i
Revised: We ﬁrst compute lower bound l i,j and upper bound u i,j of element x (cid:48) i in the set C i ( x ) :

Original: [Equation]
Revised: [Equation]

Original: a l + m l (cid:88) j = A l,i,j u i = a l − m l (cid:88) j = 
Revised: If u i,j < 0 then ReLU ( x (cid:48) i,j ) = 0 and if l i,j > 0 then ReLU ( x (cid:48) i,j ) = x (cid:48) i,j .

Original: A l,

Original: i [Equation] j If u i < 0 then ReLu ( x (cid:48)  )
Revised: In the other case where 0 is between l i,j and u i,j we deﬁne ReLU ( x (cid:48) i,j )

Original: =[-
-]
Revised: =[-
-]

Original: and if l i > 0 then ReLU ( x (cid:48) ) =
Revised: λ i,j x (cid:48) i,j + µe m i + j where e m i + j ∈ [ − 1 , 1] is a coefﬁcient for a new error term.

Original: x (cid:48) .
Revised: Formulas for λ i,j and µ i,j are the following: λ i,j = u i,j u i,j − l i,j µ i,j = − u i,j l i,j 2( u i,j − l i,j )

Original: In the other case whereis between l i and u i we deﬁne ReLU ( x (cid:48)  )
Revised: This computation can also be written in the matrix form as ReLU ( x (cid:48) i )

Original: =
Revised: =

Original: λ i  x (cid:48)
Revised: Λ i +1 x (cid:48)

Original:  + µe m l +1 where e m l +1 ∈ [ − 1 , 1] is a new coefﬁcient. Formulas for λ and µ are the following:{+M i +
+}
Revised: i + {+M i +
+}

Original: [Equation]{+e new where Λ i +
 and M i +1 are diagonal matrices with elements computed as above. Finally, new convex region C i +1 ( x ) = { a i +1 + A+}
Revised: {+e new where Λ i +
 and M i +1 are diagonal matrices with elements computed as above. Finally, new convex region C i +1 ( x ) = { a i +1 + A+}

Revised: i +1 e | e ∈ [ − 1 , 1]

Revised: m i +1 } is deﬁned as:

Revised: [Equation]

Revised: where [] denotes concatenation of matrices.

Revised: B S TATISTICAL ESTIMATION OF BOUNDS USING RANDOM PROJECTIONS

Revised: Here we describe how we apply random projection approach from Wong et al.

Revised: (2018) to estimate the bounds during training.

Revised: While Wong et al.

Revised: (2018) operate in dual framework, their method to statistically estimate the bounds during training can also be applied in primal framework which we use in this paper.

Revised: Recall that lower and upper bound for each neuron are computed as

Revised: [Equation]

Original: This computation can be written also in the matrix form to obtain matrix Λ and vector b which would update:
Revised: Thus, we need to compute || A i || 1 , which is L 1 norm of each row in the matrix A i .

Revised: Using the method from Wong et al.

Revised: (2018), based on the result from Li et al.

Revised: (2007), we estimate ||

Revised: A

Revised: i || 1 with the method of random projections.

Revised: Here A i is a matrix with d i rows and m i columns, where d i is dimensionality of the output vector in the i -th layer and m i is number of unit terms in the deﬁnition of region C i ( x ) .

Revised: The method of random projections samples standard Cauchy random matrix R of dimensions m i × k

Revised: ( k is number of projections) and then estimates || A

Revised: i || 1 ≈ median ( | A i R | ) .

Revised: To avoid computing the entire matrix

Revised: A i we substitute:

Original: [Equation]
Revised: [Equation]

Original: {+In the formula above, we set M 
+}
Revised: {+In the formula above, we set M 
+}

Revised: =

Revised: A 0 .

Revised: Now, || A

Original: {+i || 
+}
Revised: {+i || 
+}

Revised: =

Revised: || W i Λ i − 1 W i − 2 Λ

Revised: i − 3 · · · M 0 || 1 + · · · + || W i Λ i − 1 W i − 2 M

Original: {+i − 3 || 
+}
Revised: {+i − 3 || 
+}

Revised: + || W

Revised: i M i − 1 ||

Revised: [Table]

Revised: To calculate A i R we split R

Revised: =

Revised: [ R 0 , R 2 , ..., R i − 1 ] and compute:

Revised: [Equation]

Revised: Crucially, each summand can now be efﬁciently computed due to the associativity of matrix multiplication by performing the multiplication backwards.

Revised: C A DDITIONAL EXPERIMENTAL RESULTS

Revised: In this section we present additional results on SVHN and MNIST datasets.

Revised: SVHN We evaluated on SVHN dataset.

Revised: For this experiment, we used convolutional network with 2 convolutional layers of kernel size 4 and stride 1 with 32 and 128 ﬁlters respectively.

Revised: These convolutional layers are followed by a fully connected layer with 200 neurons.

Revised: Each of the layers is followed by a ReLU activation function.

Revised: For our training, we started with (cid:15) perturbation 10% higher than the one we are certifying and decreased it by 5% when progressing to the next layer.

Revised: We trained each layer for 50 epochs and used L 1 regularization factor 0.0001.

Revised: Results of this experiment are shown in Table 3.

Revised: We certiﬁed ﬁrst 1 000 images in SVHN test dataset.

Revised: Our network has both, higher accuracy and higher certiﬁed robustness, than networks trained using other techniques for provable defense.

Revised: MNIST

Revised: In the next experiment, we evaluted on MNIST dataset.

Revised: For this experiment, we used convolutional network with 2 convolutional layers with kernel sizes 5 and 4, and strides 2 followed by 1 fully connected layer.

Revised: Each of the layers is followed by a ReLU activation function.

Revised: For our training, we started with (cid:15) perturbation 10% higher than the one we are certifying and decreased it by 5% when progressing to the next layer.

Revised: We trained each layer for 50 epochs and used Lregularization factor 0.00001.

Revised: We certiﬁed ﬁrst 1 000 images in MNIST test dataset.

Revised: For perturbation 0.1, convolutional layers have ﬁlter sizes 32 and 64, and fully connected layer has 150 neurons.

Revised: Results are presented in Table 4.

Revised: Here, our numbers are comparable to those of state-of-the-art approaches.

Revised: For perturbation 0.3, convolutional layers have ﬁlter sizes 32 and 128, and fully connected layer has 400 neurons.

Revised: Results are presented in Table 4.

Revised: Here, our certiﬁed robustness is somewhat lower than state-of-the-art.

Revised: We believe this is due to the imprecise estimates of lower and upper bund via random projections.

Revised: This is also reﬂected in relatively poor performance of Wong et al.

Revised: (2018) who also rely on the same statistical estimates.

Revised: Thus, for MNIST dataset and perturbation 0.3 it is likely necessary to use exact propagation instead of the estimates.

Revised: However, this also induces large cost to the runtime.

Revised: D R EGULARIZATION

Revised: During training, we use two regularization mechanisms to make our convex relaxation tighter, both previosuly proposed in Xiao et al.

Revised: First, we use L 1 -norm regularization which is known to induce sparsity in the weights of the network.

Revised: Xiao et al.

Revised: (2019) has shown that weight sparsity helps induce more stable ReLU units which in turn makes our convex relaxation tighter (as for stable ReLU units it is already precise).

Revised: [Table]

Revised: [Table]

Revised: Second, in the i -th phase of training, we explicitly introduce a loss based on the volume of convex region C

Revised: i +1 .

Revised: To make the relaxation tighter and minimize the volume, for each neuron j in layer i + 1 we add a loss of the form max(0 , − l i +1 ,j ) max(0 , u i +1 ,j ) .

Revised: This loss corresponds to the area under ReLU relaxation, see e.g. Singh et al.

Revised: (2018) for a derivation.