Original: A DVERSARIAL V ULNERABILITY OF N EURAL N ETWORKS I NCREASES WITH I NPUT D IMENSION
Revised: A DVERSARIAL V ULNERABILITY OF N EURAL N ETWORKS I NCREASES WITH I NPUT D IMENSION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Over the past four years, neural networks have been proven vulnerable to adversarial images: targeted but imperceptible image perturbations lead to drastically different predictions.
Revised: Over the past four years, neural networks have been proven vulnerable to adversarial images: targeted but imperceptible image perturbations lead to drastically different predictions.

Original: We show that adversarial vulnerability increases with the gradients of the training objective when viewed as a function of the inputs.
Revised: We show that adversarial vulnerability increases with the gradients of the training objective when viewed as a function of the inputs.

Original: For most current network architectures, we prove that the (cid:96) 1 -norm of these gradients grows as the square root of the input size.
Revised: For most current network architectures, we prove that the (cid:96) 1 -norm of these gradients grows as the square root of the input size.

Original: These nets therefore become increasingly vulnerable with growing image size.
Revised: These nets therefore become increasingly vulnerable with growing image size.

Original: Our proofs rely on the network’s weight distribution at initialization, but extensive experiments conﬁrm that our conclusions still hold after  training.
Revised: Our proofs rely on the network’s weight distribution at initialization, but extensive experiments conﬁrm that our conclusions still hold after usual training.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Following the work of Goodfellow et al.
Revised: Following the work of Goodfellow et al.

Original: , Convolutional Neural Networks (CNNs) have been found vulnerable to adversarial examples: an adversary can drive the performance of state-of-the art
Revised: , Convolutional Neural Networks (CNNs) have been found vulnerable to adversarial examples: an adversary can drive the performance of state-of-the art

Original: CNNs down to chance level with imperceptible changes of the inputs.
Revised: CNNs down to chance level with imperceptible changes of the inputs.

Original: A number of studies have tried to address this issue, but only few have stressed that, because adversarial examples are essentially small input changes that create large output variations, they are inherently caused by large gradients of the neural network with respect to its inputs.
Revised: A number of studies have tried to address this issue, but only few have stressed that, because adversarial examples are essentially small input changes that create large output variations, they are inherently caused by large gradients of the neural network with respect to its inputs.

Original: Of course, this view, which we will focus on here, assumes that the network and loss are differentiable.
Revised: Of course, this view, which we will focus on here, assumes that the network and loss are differentiable.

Original: It has the advantage to yield a large body of speciﬁc mathematical tools, but might not be easily extendable to masked gradients, non-smooth models or the 0-1-loss.
Revised: It has the advantage to yield a large body of speciﬁc mathematical tools, but might not be easily extendable to masked gradients, non-smooth models or the 0-1-loss.

Original: Nevertheless, our conclusions might even hold for non-smooth models, given that the latter can often be viewed as smooth at a coarser level.
Revised: Nevertheless, our conclusions might even hold for non-smooth models, given that the latter can often be viewed as smooth at a coarser level.

Original: Contributions.
Revised: Contributions.

Original: More speciﬁcally, we provide theoretical and empirical arguments supporting the existence of a monotonic relationship between the gradient norm of the training objective (of a differentiable classiﬁer) and its adversarial vulnerability.
Revised: More speciﬁcally, we provide theoretical and empirical arguments supporting the existence of a monotonic relationship between the gradient norm of the training objective (of a differentiable classiﬁer) and its adversarial vulnerability.

Original: Evaluating this norm based on the weight statistics at initialization, we show that CNNs and most feed-forward networks, by design , exhibit increasingly large gradients with input dimension d , almost independently of their architecture.
Revised: Evaluating this norm based on the weight statistics at initialization, we show that CNNs and most feed-forward networks, by design , exhibit increasingly large gradients with input dimension d , almost independently of their architecture.

Original: That leaves them increasingly vulnerable to adversarial noise.
Revised: That leaves them increasingly vulnerable to adversarial noise.

Original: We corroborate our theoretical results by extensive experiments.
Revised: We corroborate our theoretical results by extensive experiments.

Original: Although some of those experiments involve adversarial regularization schemes, our goal is not to advocate a new adversarial defense (these schemes are already known), but to show how their effect can be explained by our ﬁrst order analysis.
Revised: Although some of those experiments involve adversarial regularization schemes, our goal is not to advocate a new adversarial defense (these schemes are already known), but to show how their effect can be explained by our ﬁrst order analysis.

Original: We do not claim to explain all aspects of adversarial vulnerability, but we claim that our ﬁrst order argument sufﬁces to explain a signiﬁcant part of the empirical ﬁndings on adversarial vulnerability.
Revised: We do not claim to explain all aspects of adversarial vulnerability, but we claim that our ﬁrst order argument sufﬁces to explain a signiﬁcant part of the empirical ﬁndings on adversarial vulnerability.

Original: This calls for researching the design of neural network architectures with inherently smaller gradients and provides useful guidelines to practitioners and network designers.
Revised: This calls for researching the design of neural network architectures with inherently smaller gradients and provides useful guidelines to practitioners and network designers.

Original:  F ROM A DVERSARIAL E XAMPLES TO L ARGE G RADIENTS
Revised:  F ROM A DVERSARIAL E XAMPLES TO L ARGE G RADIENTS

Original: Suppose that a given classiﬁer ϕ classiﬁes an image x as being in category ϕ
Revised: Suppose that a given classiﬁer ϕ classiﬁes an image x as being in category ϕ

Original: ( x ) .
Revised: ( x ) .

Original: An adversarial image is a small modiﬁcation of x , barely noticeable to the human eye, that sufﬁces to fool the classiﬁer into predicting a class different from ϕ ( x ) .
Revised: An adversarial image is a small modiﬁcation of x , barely noticeable to the human eye, that sufﬁces to fool the classiﬁer into predicting a class different from ϕ ( x ) .

Original: It is a small perturbation of the inputs, that creates a large variation of outputs.
Revised: It is a small perturbation of the inputs, that creates a large variation of outputs.

Original: Adversarial examples thus seem inherently related to large gradients of the network.
Revised: Adversarial examples thus seem inherently related to large gradients of the network.

Original: A connection, that we will now clarify.
Revised: A connection, that we will now clarify.

Original: Note that visible adversarial examples sometimes appear in the literature, but we deliberately focus on imperceptible ones.
Revised: Note that visible adversarial examples sometimes appear in the literature, but we deliberately focus on imperceptible ones.

Original: Adversarial vulnerability and adversarial damage.
Revised: Adversarial vulnerability and adversarial damage.

Original: In practice, an adversarial image is constructed by adding a perturbation δ to the original image x such that (cid:107) δ (cid:107) ≤ (cid:15) for some (small) number (cid:15) and a given norm (cid:107)·(cid:107) over the input space.
Revised: In practice, an adversarial image is constructed by adding a perturbation δ to the original image x such that (cid:107) δ (cid:107) ≤ (cid:15) for some (small) number (cid:15) and a given norm (cid:107)·(cid:107) over the input space.

Original: We call the perturbed input x + δ an (cid:15) -sized
Revised: We call the perturbed input x + δ an (cid:15) -sized

Original: (cid:107)·(cid:107) -attack
Revised: (cid:107)·(cid:107) -attack

Original: and say that the attack was successful when ϕ ( x + δ ) (cid:54) =
Revised: and say that the attack was successful when ϕ ( x + δ ) (cid:54) =

Original: ϕ ( x ) .
Revised: ϕ ( x ) .

Original: This motivates
Revised: This motivates

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Given a distribution P over the input-space, we call adversarial vulnerability of a classiﬁer ϕ to an (cid:15) -sized
Revised: Given a distribution P over the input-space, we call adversarial vulnerability of a classiﬁer ϕ to an (cid:15) -sized

Original: (cid:107)·(cid:107)
Revised: (cid:107)·(cid:107)

Original: -attack the probability that there exists a perturbation δ of x such that
Revised: -attack the probability that there exists a perturbation δ of x such that

Original: [Equation]
Revised: [Equation]

Original: We call the average increase-after-attack E x ∼ P [∆ L ] of a loss L the ( L -) adversarial damage (of the classiﬁer ϕ to an (cid:15) -sized
Revised: We call the average increase-after-attack E x ∼ P [∆ L ] of a loss L the ( L -) adversarial damage (of the classiﬁer ϕ to an (cid:15) -sized

Original: (cid:107)·(cid:107) -attack).
Revised: (cid:107)·(cid:107) -attack).

Original: When L is the 0-1-loss L 0 / 1 , adversarial damage is the accuracy-drop after attack.
Revised: When L is the 0-1-loss L 0 / 1 , adversarial damage is the accuracy-drop after attack.

Original: The 0-1-loss damage is always smaller than adversarial vulnerability, because vulnerability counts all class-changes of ϕ ( x ) , whereas some of them may be neutral to adversarial damage (e.g. a change between two wrong classes).
Revised: The 0-1-loss damage is always smaller than adversarial vulnerability, because vulnerability counts all class-changes of ϕ ( x ) , whereas some of them may be neutral to adversarial damage (e.g. a change between two wrong classes).

Original: The L 0 / 1 -adversarial damage thus lower bounds adversarial vulnerability.
Revised: The L 0 / 1 -adversarial damage thus lower bounds adversarial vulnerability.

Original: Both are even equal when the classiﬁer is perfect (before attack), because then every change of label introduces an error.
Revised: Both are even equal when the classiﬁer is perfect (before attack), because then every change of label introduces an error.

Original: It is hence tempting to evaluate adversarial vulnerability with L 0 / 1 -adversarial damage.
Revised: It is hence tempting to evaluate adversarial vulnerability with L 0 / 1 -adversarial damage.

Original: From ∆ L 0 / 1 to ∆ L and to ∂ x L .
Revised: From ∆ L 0 / 1 to ∆ L and to ∂ x L .

Original: In practice however, we do not train our classiﬁers with the non-differentiable 0-1-loss but use a smoother loss L , such as the cross-entropy loss.
Revised: In practice however, we do not train our classiﬁers with the non-differentiable 0-1-loss but use a smoother loss L , such as the cross-entropy loss.

Original: For similar reasons, we will now investigate the adversarial damage E x [∆ L ( x , c )] with loss L rather than L 0 / 1 .
Revised: For similar reasons, we will now investigate the adversarial damage E x [∆ L ( x , c )] with loss L rather than L 0 / 1 .

Original: Like for Goodfellow et al.
Revised: Like for Goodfellow et al.

Original: ; Lyu et al. (2015); Sinha et al.
Revised: ; Lyu et al. (2015); Sinha et al.

Original: (2018) and many others, a classiﬁer ϕ will hence be robust if, on average over x , a small adversarial perturbation δ of x creates only a small variation δ L of the loss.
Revised: (2018) and many others, a classiﬁer ϕ will hence be robust if, on average over x , a small adversarial perturbation δ of x creates only a small variation δ L of the loss.

Original: Now, if (cid:107) δ (cid:107) ≤ (cid:15) , then a ﬁrst order Taylor expansion in (cid:15) shows that
Revised: Now, if (cid:107) δ (cid:107) ≤ (cid:15) , then a ﬁrst order Taylor expansion in (cid:15) shows that

Original: [Equation]
Revised: [Equation]

Original: where ∂ x L denotes the gradient of L with respect to x , and where the last equality stems from the deﬁnition of the dual norm |||·||| of (cid:107)·(cid:107) .
Revised: where ∂ x L denotes the gradient of L with respect to x , and where the last equality stems from the deﬁnition of the dual norm |||·||| of (cid:107)·(cid:107) .

Original: Now two remarks.
Revised: Now two remarks.

Original: First: the dual norm only kicks in because we let the input noise δ optimally adjust to the coordinates of ∂ x
Revised: First: the dual norm only kicks in because we let the input noise δ optimally adjust to the coordinates of ∂ x

Original: L within its (cid:15) -constraint.
Revised: L within its (cid:15) -constraint.

Original: This is the brand mark of adversarial noise: the different coordinates add up, instead of statistically canceling each other out as they would with random noise.
Revised: This is the brand mark of adversarial noise: the different coordinates add up, instead of statistically canceling each other out as they would with random noise.

Original: For example, if we impose that (cid:107) δ (cid:107) 2 ≤ (cid:15) , then δ will strictly align with ∂ x L .
Revised: For example, if we impose that (cid:107) δ (cid:107) 2 ≤ (cid:15) , then δ will strictly align with ∂ x L .

Original: If instead (cid:107) δ (cid:107) ∞ ≤ (cid:15) , then δ will align with the sign of the coordinates of ∂ x L
Revised: If instead (cid:107) δ (cid:107) ∞ ≤ (cid:15) , then δ will align with the sign of the coordinates of ∂ x L

Original: Second remark: while the Taylor expansion in (2) becomes exact for inﬁnitesimal perturbations, for ﬁnite ones it may actually be dominated by higher-order terms.
Revised: Second remark: while the Taylor expansion in (2) becomes exact for inﬁnitesimal perturbations, for ﬁnite ones it may actually be dominated by higher-order terms.

Original: Our experiments (Figures 1 & 2) however strongly suggest that in practice the ﬁrst order term dominates the others.
Revised: Our experiments (Figures 1 & 2) however strongly suggest that in practice the ﬁrst order term dominates the others.

Original: Now, remembering that the dual norm of an (cid:96) p -norm is the corresponding (cid:96) q -norm, and summarizing, we have proven
Revised: Now, remembering that the dual norm of an (cid:96) p -norm is the corresponding (cid:96) q -norm, and summarizing, we have proven

Original: Lemma 2.
Revised: Lemma 2.

Original: At ﬁrst order approximation in (cid:15) , an (cid:15) -sized adversarial attack generated with norm (cid:107)·(cid:107) increases the loss L at point x by (cid:15) ||| ∂ x L||| , where |||·||| is the dual norm of (cid:107)·(cid:107) .
Revised: At ﬁrst order approximation in (cid:15) , an (cid:15) -sized adversarial attack generated with norm (cid:107)·(cid:107) increases the loss L at point x by (cid:15) ||| ∂ x L||| , where |||·||| is the dual norm of (cid:107)·(cid:107) .

Original: In particular, an (cid:15) -sized (cid:96) p -attack increases the loss by (cid:15)
Revised: In particular, an (cid:15) -sized (cid:96) p -attack increases the loss by (cid:15)

Original: [Equation]
Revised: [Equation]

Original: q where 1 ≤ p ≤ ∞ and 1 p + 1 q = 1 .
Revised: q where 1 ≤ p ≤ ∞ and 1 p + 1 q = 1 .

Original: Consequently, the adversarial damage of a classiﬁer with loss L to (cid:15) -sized
Revised: Consequently, the adversarial damage of a classiﬁer with loss L to (cid:15) -sized

Original: attacks generated with norm (cid:107)·(cid:107) is (cid:15) E x ||| ∂ x L||| .
Revised: attacks generated with norm (cid:107)·(cid:107) is (cid:15) E x ||| ∂ x L||| .

Original: This is valid only at ﬁrst order, but it proves that at least this kind of ﬁrst-order vulnerability is present.
Revised: This is valid only at ﬁrst order, but it proves that at least this kind of ﬁrst-order vulnerability is present.

Original: We will see that the ﬁrst-order predictions closely match the experiments, and that this insight helps protecting even against iterative (non-ﬁrst-order) attack methods (Figure 1).
Revised: We will see that the ﬁrst-order predictions closely match the experiments, and that this insight helps protecting even against iterative (non-ﬁrst-order) attack methods (Figure 1).

Original: Calibrating the threshold (cid:15) to the attack-norm (cid:107)·(cid:107) .
Revised: Calibrating the threshold (cid:15) to the attack-norm (cid:107)·(cid:107) .

Original: Lemma 2 shows that adversarial vulnerability depends on three main factors: (i) (cid:107)·(cid:107) , the norm chosen for the attack (ii) (cid:15) , the size of the attack, and (iii) E x ||| ∂ x L||| , the expected dual norm of ∂ x L .
Revised: Lemma 2 shows that adversarial vulnerability depends on three main factors: (i) (cid:107)·(cid:107) , the norm chosen for the attack (ii) (cid:15) , the size of the attack, and (iii) E x ||| ∂ x L||| , the expected dual norm of ∂ x L .

Original: We could see Point (i) as a measure of our sensibility to image perturbations, (ii) as our sensibility threshold, and (iii) as the classiﬁer’s expected marginal sensibility to a unit perturbation.
Revised: We could see Point (i) as a measure of our sensibility to image perturbations, (ii) as our sensibility threshold, and (iii) as the classiﬁer’s expected marginal sensibility to a unit perturbation.

Original: E x ||| ∂ x L||| hence intuitively captures the discrepancy between our perception (as modeled by (cid:107)·(cid:107) ) and the classiﬁer’s perception for an input-perturbation of small size (cid:15) .
Revised: E x ||| ∂ x L||| hence intuitively captures the discrepancy between our perception (as modeled by (cid:107)·(cid:107) ) and the classiﬁer’s perception for an input-perturbation of small size (cid:15) .

Original: Of course, this viewpoint supposes that we actually found a norm (cid:107)·(cid:107) (or more generally a metric) that faithfully reﬂects human perception – a project in its own right, far beyond the scope of this paper.
Revised: Of course, this viewpoint supposes that we actually found a norm (cid:107)·(cid:107) (or more generally a metric) that faithfully reﬂects human perception – a project in its own right, far beyond the scope of this paper.

Original: However, it is clear that the threshold (cid:15) that we choose should depend on the norm (cid:107)·(cid:107) and hence on the input-dimension d .
Revised: However, it is clear that the threshold (cid:15) that we choose should depend on the norm (cid:107)·(cid:107) and hence on the input-dimension d .

Original: In particular, for a given pixel-wise order of magnitude of the perturbations δ , the (cid:96) p -norm of the perturbation will scale like d 1 /p .
Revised: In particular, for a given pixel-wise order of magnitude of the perturbations δ , the (cid:96) p -norm of the perturbation will scale like d 1 /p .

Original: This suggests to write the threshold (cid:15) p used with (cid:96) p -attacks as:
Revised: This suggests to write the threshold (cid:15) p used with (cid:96) p -attacks as:

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) ∞ denotes a dimension-independent constant.
Revised: where (cid:15) ∞ denotes a dimension-independent constant.

Original: In Appendix D we show that this scaling also preserves the average signal-to-noise ratio (cid:107) x (cid:107) 2 / (cid:107) δ (cid:107) 2 , both across norms and dimensions, so that (cid:15) p could correspond to a constant human perception-threshold.
Revised: In Appendix D we show that this scaling also preserves the average signal-to-noise ratio (cid:107) x (cid:107) 2 / (cid:107) δ (cid:107) 2 , both across norms and dimensions, so that (cid:15) p could correspond to a constant human perception-threshold.

Original: With this in mind, the impatient reader may already jump to Section 3, which contains our main contributions: the estimation of E x (cid:107) ∂ x L(cid:107) q for standard feed-forward nets.
Revised: With this in mind, the impatient reader may already jump to Section 3, which contains our main contributions: the estimation of E x (cid:107) ∂ x L(cid:107) q for standard feed-forward nets.

Original: Meanwhile, the rest of this section shortly discusses two straightforward defenses that we will use later and that further illustrate the role of gradients.
Revised: Meanwhile, the rest of this section shortly discusses two straightforward defenses that we will use later and that further illustrate the role of gradients.

Original: A new old regularizer.
Revised: A new old regularizer.

Original: Lemma 2 shows that the loss of the network after an (cid:15) 2 -sized (cid:107)·(cid:107) -attack is
Revised: Lemma 2 shows that the loss of the network after an (cid:15) 2 -sized (cid:107)·(cid:107) -attack is

Original: [Equation]
Revised: [Equation]

Original: It is thus natural to take this loss-after-attack as a new training objective.
Revised: It is thus natural to take this loss-after-attack as a new training objective.

Original: Here we introduced a factorfor reasons that will become clear in a moment.
Revised: Here we introduced a factorfor reasons that will become clear in a moment.

Original: Incidentally, for (cid:107)·(cid:107) =
Revised: Incidentally, for (cid:107)·(cid:107) =

Original: (cid:107)·(cid:107) 2 , this new loss reduces to an old regularization-scheme proposed by Drucker & LeCun (1991) called double-backpropagation .
Revised: (cid:107)·(cid:107) 2 , this new loss reduces to an old regularization-scheme proposed by Drucker & LeCun (1991) called double-backpropagation .

Original: At the time, the authors argued that slightly decreasing a function’s or a classiﬁer’s sensitivity to input perturbations should improve generalization.
Revised: At the time, the authors argued that slightly decreasing a function’s or a classiﬁer’s sensitivity to input perturbations should improve generalization.

Original: In a sense, this is exactly our motivation when defending against adversarial examples.
Revised: In a sense, this is exactly our motivation when defending against adversarial examples.

Original: It is thus not surprising to end up with the same regularization term.
Revised: It is thus not surprising to end up with the same regularization term.

Original: Note that our reasoning only shows that training with one speciﬁc norm |||·||| in (4) helps to protect against adversarial examples generated from (cid:107)·(cid:107) .
Revised: Note that our reasoning only shows that training with one speciﬁc norm |||·||| in (4) helps to protect against adversarial examples generated from (cid:107)·(cid:107) .

Original: A priori, we do not know what will happen for attacks generated with other norms; but our experiments suggest that training with one norm also protects against other attacks (see Figure 2 and Section 4.1).
Revised: A priori, we do not know what will happen for attacks generated with other norms; but our experiments suggest that training with one norm also protects against other attacks (see Figure 2 and Section 4.1).

Original: Link to adversarially-augmented training.
Revised: Link to adversarially-augmented training.

Original: In (1), (cid:15) designates an attack-size threshold, while in (4), it is a regularization-strength.
Revised: In (1), (cid:15) designates an attack-size threshold, while in (4), it is a regularization-strength.

Original: Rather than a notation conﬂict, this reﬂects an intrinsic duality between two complementary interpretations of (cid:15) , which we now investigate further.
Revised: Rather than a notation conﬂict, this reﬂects an intrinsic duality between two complementary interpretations of (cid:15) , which we now investigate further.

Original: Suppose that, instead of using the loss-after-attack, we augment our training set with (cid:15) -sized
Revised: Suppose that, instead of using the loss-after-attack, we augment our training set with (cid:15) -sized

Original: (cid:107)·(cid:107) -attacks x + δ , where for each training point x , the perturbation δ is generated on the ﬂy to locally maximize the loss-increase.
Revised: (cid:107)·(cid:107) -attacks x + δ , where for each training point x , the perturbation δ is generated on the ﬂy to locally maximize the loss-increase.

Original: Then we are effectively training with
Revised: Then we are effectively training with

Original: [Equation]
Revised: [Equation]

Original: where by construction δ satisﬁes (2).
Revised: where by construction δ satisﬁes (2).

Original: We will refer to this technique as adversarially augmented training .
Revised: We will refer to this technique as adversarially augmented training .

Original: It was ﬁrst introduced by Goodfellow et al.
Revised: It was ﬁrst introduced by Goodfellow et al.

Original: (2015) with (cid:107)·(cid:107) =
Revised: (2015) with (cid:107)·(cid:107) =

Original: (cid:107)·(cid:107) ∞ under the name of
Revised: (cid:107)·(cid:107) ∞ under the name of

Original: FGSM 1 -augmented training.
Revised: FGSM 1 -augmented training.

Original: Using the ﬁrst order Taylor expansion in (cid:15) of (2), this ‘old-plus-postattack’ loss of (5) simply reduces to our loss-after-attack, which proves
Revised: Using the ﬁrst order Taylor expansion in (cid:15) of (2), this ‘old-plus-postattack’ loss of (5) simply reduces to our loss-after-attack, which proves

Original: Proposition 3.
Revised: Proposition 3.

Original: Up to ﬁrst-order approximations in (cid:15) , ˜ L (cid:15), (cid:107)·(cid:107)
Revised: Up to ﬁrst-order approximations in (cid:15) , ˜ L (cid:15), (cid:107)·(cid:107)

Original: =
Revised: =

Original: L (cid:15), |||·||| .
Revised: L (cid:15), |||·||| .

Original: Said differently, for small enough (cid:15) , adversarially-augmented training with (cid:15) -sized
Revised: Said differently, for small enough (cid:15) , adversarially-augmented training with (cid:15) -sized

Original: (cid:107)·(cid:107) -attacks amounts to penalizing the dual norm |||·||| of ∂ x L with weight (cid:15)/ 2 .
Revised: (cid:107)·(cid:107) -attacks amounts to penalizing the dual norm |||·||| of ∂ x L with weight (cid:15)/ 2 .

Original: In particular, double-backpropagation corresponds to training with (cid:96) 2 -attacks, while FGSM-augmented training corresponds to an (cid:96) 1 -penalty on ∂ x L .
Revised: In particular, double-backpropagation corresponds to training with (cid:96) 2 -attacks, while FGSM-augmented training corresponds to an (cid:96) 1 -penalty on ∂ x L .

Original: This correspondence between training with perturbations and using a regularizer can be compared to Tikhonov regularization:
Revised: This correspondence between training with perturbations and using a regularizer can be compared to Tikhonov regularization:

Original: Tikhonov regularization amounts to training with random noise Bishop (1995), while training with adversarial noise amounts to penalizing ∂ x L .
Revised: Tikhonov regularization amounts to training with random noise Bishop (1995), while training with adversarial noise amounts to penalizing ∂ x L .

Original: Section 4.1 veriﬁes the correspondence between adversarial augmentation and gradient regularization empirically, which also strongly suggests the empirical validity of the ﬁrst-order Taylor expansion in (2).
Revised: Section 4.1 veriﬁes the correspondence between adversarial augmentation and gradient regularization empirically, which also strongly suggests the empirical validity of the ﬁrst-order Taylor expansion in (2).

Original:  E STIMATING (cid:107) ∂ x L(cid:107) q TO E VALUATE A DVERSARIAL V ULNERABILITY
Revised:  E STIMATING (cid:107) ∂ x L(cid:107) q TO E VALUATE A DVERSARIAL V ULNERABILITY

Original: In this section, we evaluate the size of (cid:107) ∂ x L(cid:107) q for standard neural network architectures.
Revised: In this section, we evaluate the size of (cid:107) ∂ x L(cid:107) q for standard neural network architectures.

Original: We start with fully-connected networks, and ﬁnish with a much more general theorem that, not only encompasses CNNs (with or without strided convolutions), but also shows that the gradient-norms are essentially independent of the network topology.
Revised: We start with fully-connected networks, and ﬁnish with a much more general theorem that, not only encompasses CNNs (with or without strided convolutions), but also shows that the gradient-norms are essentially independent of the network topology.

Original: We start our analysis by showing how changing q affects the size of (cid:107) ∂ x L(cid:107) q .
Revised: We start our analysis by showing how changing q affects the size of (cid:107) ∂ x L(cid:107) q .

Original: Suppose for a moment that the coordinates of ∂ x L have typical magnitude | ∂ x L| .
Revised: Suppose for a moment that the coordinates of ∂ x L have typical magnitude | ∂ x L| .

Original: Then (cid:107) ∂ x L(cid:107) q scales like d 1 /q |
Revised: Then (cid:107) ∂ x L(cid:107) q scales like d 1 /q |

Original: ∂ x L| .
Revised: ∂ x L| .

Original: Consequently
Revised: Consequently

Original: [Equation]
Revised: [Equation]

Original: This equation carries two important messages.
Revised: This equation carries two important messages.

Original: First, we see how (cid:107) ∂ x L(cid:107) q depends on d and q .
Revised: First, we see how (cid:107) ∂ x L(cid:107) q depends on d and q .

Original: The dependence seems highest for q = 1 .
Revised: The dependence seems highest for q = 1 .

Original: But once we account for the varying perceptibility threshold (cid:15) p ∝ d 1 /p , we see that adversarial vulnerability scales like d · | ∂ x L| , whatever (cid:96) p -norm we use.
Revised: But once we account for the varying perceptibility threshold (cid:15) p ∝ d 1 /p , we see that adversarial vulnerability scales like d · | ∂ x L| , whatever (cid:96) p -norm we use.

Original: Second, (6) shows that to be robust against any type of (cid:96) p -attack at any input-dimension d , the average absolute value of the coefﬁcients of ∂ x L must grow slower than 1 /d .
Revised: Second, (6) shows that to be robust against any type of (cid:96) p -attack at any input-dimension d , the average absolute value of the coefﬁcients of ∂ x L must grow slower than 1 /d .

Original: Now, here is the catch, which brings us to our core insight.
Revised: Now, here is the catch, which brings us to our core insight.

Original: 3. C ORE I DEA : O NE N EURON WITH M ANY I NPUTS
Revised: 3. C ORE I DEA : O NE N EURON WITH M ANY I NPUTS

Original: In order to preserve the activation variance of the neurons from layer to layer, the neural weights are usually initialized with a variance that is inversely proportional to the number of inputs per neuron.
Revised: In order to preserve the activation variance of the neurons from layer to layer, the neural weights are usually initialized with a variance that is inversely proportional to the number of inputs per neuron.

Original: Imagine for a moment that the network consisted only of one output neuron o linearly connected to all input pixels.
Revised: Imagine for a moment that the network consisted only of one output neuron o linearly connected to all input pixels.

Original: For the purpose of this example, we assimilate o and L .
Revised: For the purpose of this example, we assimilate o and L .

Original: Because we initialize the weights with a variance of 1 /d , their average absolute value | ∂ x o | ≡ | ∂ x L| grows like 1 / √ d , rather than the required 1 /d .
Revised: Because we initialize the weights with a variance of 1 /d , their average absolute value | ∂ x o | ≡ | ∂ x L| grows like 1 / √ d , rather than the required 1 /d .

Original: By (6), the adversarial vulnerability (cid:15) (cid:107)
Revised: By (6), the adversarial vulnerability (cid:15) (cid:107)

Original: [Equation]
Revised: [Equation]

Original: ∂ x L(cid:107) q therefore increases like d/ √ d = √ d .
Revised: ∂ x L(cid:107) q therefore increases like d/ √ d = √ d .

Original: This toy example shows that the standard initialization scheme, which preserves the variance from layer to layer, causes the average coordinate-size | ∂ x L| to grow like 1 / √ d instead of 1 /d .
Revised: This toy example shows that the standard initialization scheme, which preserves the variance from layer to layer, causes the average coordinate-size | ∂ x L| to grow like 1 / √ d instead of 1 /d .

Original: When an (cid:96) ∞ -attack tweaks its (cid:15) -sized input-perturbations to align with the coordinate-signs of ∂ x L , all coordinates of ∂ x L add up in absolute value, resulting in an output-perturbation that scales like (cid:15) √ d and leaves the network increasingly vulnerable with growing input-dimension.
Revised: When an (cid:96) ∞ -attack tweaks its (cid:15) -sized input-perturbations to align with the coordinate-signs of ∂ x L , all coordinates of ∂ x L add up in absolute value, resulting in an output-perturbation that scales like (cid:15) √ d and leaves the network increasingly vulnerable with growing input-dimension.

Original: 3. G ENERALIZATION TO D EEP N ETWORKS
Revised: 3. G ENERALIZATION TO D EEP N ETWORKS

Original: Our next theorems generalize the previous toy example to a very wide class of feedforward nets with
Revised: Our next theorems generalize the previous toy example to a very wide class of feedforward nets with

Original: ReLU activation functions.
Revised: ReLU activation functions.

Original: For illustration purposes, we start with fully connected nets and only then proceed to the broader class, which includes any succession of (possibly strided) convolutional layers.
Revised: For illustration purposes, we start with fully connected nets and only then proceed to the broader class, which includes any succession of (possibly strided) convolutional layers.

Original: In essence, the proofs iterate our insight on one layer over a sequence of layers.
Revised: In essence, the proofs iterate our insight on one layer over a sequence of layers.

Original: They all rely on the following set ( H ) of hypotheses:
Revised: They all rely on the following set ( H ) of hypotheses:

Original: H1 Non-input neurons are followed by a ReLU killing half of its inputs, independently of the weights.
Revised: H1 Non-input neurons are followed by a ReLU killing half of its inputs, independently of the weights.

Original: H2 Neurons are partitioned into layers, meaning groups that each path traverses at most once.
Revised: H2 Neurons are partitioned into layers, meaning groups that each path traverses at most once.

Original: H3 All weights have 0 expectation and variance 2 / ( in-degree ) (‘He-initialization’).
Revised: H3 All weights have 0 expectation and variance 2 / ( in-degree ) (‘He-initialization’).

Original: H

Revised: H

Original: The weights from different layers are independent.
Revised: The weights from different layers are independent.

Original: H5 Two distinct weights w, w (cid:48) from a same node satisfy E [ w w (cid:48) ]
Revised: H5 Two distinct weights w, w (cid:48) from a same node satisfy E [ w w (cid:48) ]

Original: = 0 .
Revised: = 0 .

Original: If we follow common practice and initialize our nets as proposed by He et al. (2015), then H3-H5 are satisﬁed at initialization by design, while H1 is usually a very good approximation (Balduzzi et al., 2017).
Revised: If we follow common practice and initialize our nets as proposed by He et al. (2015), then H3-H5 are satisﬁed at initialization by design, while H1 is usually a very good approximation (Balduzzi et al., 2017).

Original: Note that such i.i.d.
Revised: Note that such i.i.d.

Original: weight assumptions have been widely used to analyze neural nets and are at the heart of very inﬂuential and successful prior work (e.g., equivalence between neural nets and Gaussian processes as pioneered by Neal 1996).
Revised: weight assumptions have been widely used to analyze neural nets and are at the heart of very inﬂuential and successful prior work (e.g., equivalence between neural nets and Gaussian processes as pioneered by Neal 1996).

Original: Nevertheless, they do not hold after training.
Revised: Nevertheless, they do not hold after training.

Original: That is why all our statements in this section are to be understood as orders of magnitudes that are very well satisﬁed at initialization in theory and in practice, and that we will conﬁrm experimentally after training in Section 4.
Revised: That is why all our statements in this section are to be understood as orders of magnitudes that are very well satisﬁed at initialization in theory and in practice, and that we will conﬁrm experimentally after training in Section 4.

Original: Said differently, while our theorems rely on the statistics of neural nets at initialization, our experiments conﬁrm their conclusions after training.
Revised: Said differently, while our theorems rely on the statistics of neural nets at initialization, our experiments conﬁrm their conclusions after training.

Original: Theorem 4 ( Vulnerability of Fully Connected Nets ) .
Revised: Theorem 4 ( Vulnerability of Fully Connected Nets ) .

Original: Consider a succession of fully connected layers with ReLU activations which takes inputs x of dimension d , satisﬁes assumptions ( H ), and outputs logits f k ( x ) that get fed to a ﬁnal cross-entropy-loss layer L .
Revised: Consider a succession of fully connected layers with ReLU activations which takes inputs x of dimension d , satisﬁes assumptions ( H ), and outputs logits f k ( x ) that get fed to a ﬁnal cross-entropy-loss layer L .

Original: Then the coordinates of ∂ x f k grow like 1 / √ d , and
Revised: Then the coordinates of ∂ x f k grow like 1 / √ d , and

Original: [Equation]
Revised: [Equation]

Original: and (cid:15)
Revised: and (cid:15)

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: These networks are thus increasingly vulnerable to (cid:96) p -attacks with growing input-dimension.
Revised: These networks are thus increasingly vulnerable to (cid:96) p -attacks with growing input-dimension.

Original: Theorem 4 is a special case of the next theorem, which will show that the previous conclusions are essentially independent of the network-topology.
Revised: Theorem 4 is a special case of the next theorem, which will show that the previous conclusions are essentially independent of the network-topology.

Original: We will use the following symmetry assumption on the neural connections.
Revised: We will use the following symmetry assumption on the neural connections.

Original: For a given path p , let the path-degree d p be the multiset of encountered in-degrees along path p .
Revised: For a given path p , let the path-degree d p be the multiset of encountered in-degrees along path p .

Original: For a fully connected network, this is the unordered sequence of layer-sizes preceding the last path-node, including the input-layer.
Revised: For a fully connected network, this is the unordered sequence of layer-sizes preceding the last path-node, including the input-layer.

Original: Now consider the multiset { d p } p ∈P ( x,o ) of all path-degrees when p varies among all paths from input x to output o .
Revised: Now consider the multiset { d p } p ∈P ( x,o ) of all path-degrees when p varies among all paths from input x to output o .

Original: The symmetry assumption (relatively to o ) is
Revised: The symmetry assumption (relatively to o ) is

Original: ( S )
Revised: ( S )

Original: All input nodes x have the same multiset { d p } p ∈P ( x,o ) of path-degrees from x to o .
Revised: All input nodes x have the same multiset { d p } p ∈P ( x,o ) of path-degrees from x to o .

Original: Intuitively, this means that the statistics of degrees encountered along paths to the output are the same for all input nodes.
Revised: Intuitively, this means that the statistics of degrees encountered along paths to the output are the same for all input nodes.

Original: This symmetry assumption is exactly satisﬁed by fully connected nets, almost satisﬁed by CNNs (up to boundary effects, which can be alleviated via periodic or mirror padding) and exactly satisﬁed by strided layers, if the layer-size is a multiple of the stride.
Revised: This symmetry assumption is exactly satisﬁed by fully connected nets, almost satisﬁed by CNNs (up to boundary effects, which can be alleviated via periodic or mirror padding) and exactly satisﬁed by strided layers, if the layer-size is a multiple of the stride.

Original: Theorem 5 ( Vulnerability of Feedforward Nets ) .
Revised: Theorem 5 ( Vulnerability of Feedforward Nets ) .

Original: Consider any feed-forward network with linear connections and ReLU activation functions.
Revised: Consider any feed-forward network with linear connections and ReLU activation functions.

Original: Assume the net satisﬁes assumptions ( H ) and outputs logits f k ( x ) that get fed to the cross-entropy-loss L .
Revised: Assume the net satisﬁes assumptions ( H ) and outputs logits f k ( x ) that get fed to the cross-entropy-loss L .

Original: Then (cid:107) ∂
Revised: Then (cid:107) ∂

Original: x f k (cid:107) 2 is independent of the input dimension d and (cid:15) 2 (cid:107)
Revised: x f k (cid:107) 2 is independent of the input dimension d and (cid:15) 2 (cid:107)

Original: ∂ x L(cid:107) 2 ∝ √ d .
Revised: ∂ x L(cid:107) 2 ∝ √ d .

Original: Moreover, if the net satisﬁes the symmetry assumption ( S ), then | ∂ x f k | ∝ 1 / √ d and (7) still holds: (cid:107) ∂ x L(cid:107) q ∝ d 1 q − 12 and (cid:15) p (cid:107) ∂ x L(cid:107) q ∝ √ d .
Revised: Moreover, if the net satisﬁes the symmetry assumption ( S ), then | ∂ x f k | ∝ 1 / √ d and (7) still holds: (cid:107) ∂ x L(cid:107) q ∝ d 1 q − 12 and (cid:15) p (cid:107) ∂ x L(cid:107) q ∝ √ d .

Original: Theorems 4 and 5 are proven in Appendix B.
Revised: Theorems 4 and 5 are proven in Appendix B.

Original: The main proof idea is that in the gradient norm computation, the He-initialization exactly compensates the combinatorics of the number of paths in the network, so that this norm becomes independent of the network topology.
Revised: The main proof idea is that in the gradient norm computation, the He-initialization exactly compensates the combinatorics of the number of paths in the network, so that this norm becomes independent of the network topology.

Original: In particular, we get
Revised: In particular, we get

Original: Corollary 6 ( Vulnerability of CNNs ) .
Revised: Corollary 6 ( Vulnerability of CNNs ) .

Original: In any succession of convolution and dense layers, strided or not, with ReLU activations, that satisﬁes assumptions ( H ) and outputs logits that get fed to the cross-entropy-loss L , the gradient of the logit-coordinates scale like 1 / √ d and (7) is satisﬁed.
Revised: In any succession of convolution and dense layers, strided or not, with ReLU activations, that satisﬁes assumptions ( H ) and outputs logits that get fed to the cross-entropy-loss L , the gradient of the logit-coordinates scale like 1 / √ d and (7) is satisﬁed.

Original: It is hence increasingly vulnerable with growing input-resolution to attacks generated with any (cid:96) p -norm.
Revised: It is hence increasingly vulnerable with growing input-resolution to attacks generated with any (cid:96) p -norm.

Original: Appendix A shows that the network gradient are dampened when replacing strided layers by average poolings, essentially because average-pooling weights do not follow the He-init assumption H3.
Revised: Appendix A shows that the network gradient are dampened when replacing strided layers by average poolings, essentially because average-pooling weights do not follow the He-init assumption H3.

Original:  E MPIRICAL R ESULTS
Revised:  E MPIRICAL R ESULTS

Original: In Section 4.1, we empirically verify the validity of the ﬁrst-order Taylor approximation made in (2) (Fig.1), for example by checking the correspondence between loss-gradient regularization and adversarially-augmented training (Fig.2).
Revised: In Section 4.1, we empirically verify the validity of the ﬁrst-order Taylor approximation made in (2) (Fig.1), for example by checking the correspondence between loss-gradient regularization and adversarially-augmented training (Fig.2).

Original: Section 4.2 then empirically veriﬁes that both the average (cid:96) 1 -norm of ∂ x L and the adversarial vulnerability grow like √ d as predicted by Corollary 6.
Revised: Section 4.2 then empirically veriﬁes that both the average (cid:96) 1 -norm of ∂ x L and the adversarial vulnerability grow like √ d as predicted by Corollary 6.

Original: For all experiments, we approximate adversarial vulnerability using various attacks of the Foolboxpackage (Rauber et al., 2017).
Revised: For all experiments, we approximate adversarial vulnerability using various attacks of the Foolboxpackage (Rauber et al., 2017).

Original: We use an (cid:96) ∞ attack-threshold of size (cid:15) ∞ = 0 .
Revised: We use an (cid:96) ∞ attack-threshold of size (cid:15) ∞ = 0 .

Original: 005  which, for pixel-values ranging from 0 to 1, is completely imperceptible but sufﬁces to fool the classiﬁers on a signiﬁcant proportion of examples.
Revised: 005 (and later 0 . 002 ) which, for pixel-values ranging from 0 to 1, is completely imperceptible but sufﬁces to fool the classiﬁers on a signiﬁcant proportion of examples.

Original: This (cid:15) ∞ - threshold should not be confused with the regularizationstrengths (cid:15) appearing in (4) and (5), which will be varied in some experiments.[Figure] PPROXIMATION [Figure]
Revised: This (cid:15) ∞ - threshold should not be confused with the regularization-strengths (cid:15) appearing in (4) and (5), which will be varied in some experiments.

Original:  G RADIENT P ENALTY 
Revised: 4. F IRST -O RDER A PPROXIMATION , G RADIENT P ENALTY , A DVERSARIAL A UGMENTATION

Original: [Figure]
Revised: [Figure]

Original: A DVERSARIAL A UGMENTATION [Figure] different attack-types for 10 trained networks as a function of ( a ) the (cid:96) 1 gradient regularization-strength (cid:15) used to train the nets and ( b ) the average gradient-norm.
Revised: Adversarial vulnerability approximated by different attack-types for 10 trained networks as a function of ( a ) the (cid:96) 1 gradient regularization-strength (cid:15) used to train the nets and ( b ) the average gradient-norm.

Original: These curves conﬁrm that the ﬁrst-order expansion term in (2) is a crucial component of adversarial vulnerability.
Revised: These curves conﬁrm that the ﬁrst-order expansion term in (2) is a crucial component of adversarial vulnerability.

Original: [Figure]
Revised: [Figure]

Original: We train several CNNs with same architecture to classify CIFAR-10 images (Krizhevsky, 2009).
Revised: We train several CNNs with same architecture to classify CIFAR-10 images (Krizhevsky, 2009).

Original: For each net, we use a speciﬁc training method with a speciﬁc regularization value (cid:15) .
Revised: For each net, we use a speciﬁc training method with a speciﬁc regularization value (cid:15) .

Original: The training methods used were (cid:96) 1 - and (cid:96) 2 -penalization of ∂ x L (Eq. 4), adversarial augmentation with (cid:96) ∞ - and (cid:96) 2 - attacks (Eq. 5), projected gradient descent (PGD) with randomized starts (7 steps per attack with step-size = .
Revised: The training methods used were (cid:96) 1 - and (cid:96) 2 -penalization of ∂ x L (Eq. 4), adversarial augmentation with (cid:96) ∞ - and (cid:96) 2 - attacks (Eq. 5), projected gradient descent (PGD) with randomized starts (7 steps per attack with step-size = .

Original: 2 (cid:15) ∞ ; see Madry et al. 2018) and the cross-Lipschitz regularizer (Eq. 17 in Appendix C).
Revised: 2 (cid:15) ∞ ; see Madry et al. 2018) and the cross-Lipschitz regularizer (Eq. 17 in Appendix C).

Original: We then test the adversarial vulnerability of each trained network using the following attack-methods: single-step (cid:96) ∞ - (FGSM) and (cid:96) 2 -attacks, iterative (cid:96) ∞ - (PGD) and (cid:96) 2 -attacks, and DeepFool attacks (Moosavi-Dezfooli et al., 2016).
Revised: We then test the adversarial vulnerability of each trained network using the following attack-methods: single-step (cid:96) ∞ - (FGSM) and (cid:96) 2 -attacks, iterative (cid:96) ∞ - (PGD) and (cid:96) 2 -attacks, and DeepFool attacks (Moosavi-Dezfooli et al., 2016).

Original: All networks have 6 ‘strided convolution → batchnorm → ReLU’ layers with strides [1, 2, 2, 2, 2, 2] respectively and 64 output-channels each, followed by a ﬁnal fully-connected linear layer.
Revised: All networks have 6 ‘strided convolution → batchnorm → ReLU’ layers with strides [1, 2, 2, 2, 2, 2] respectively and 64 output-channels each, followed by a ﬁnal fully-connected linear layer.

Original: Results are summarized in Figures 1 and 2.
Revised: Results are summarized in Figures 1 and 2.

Original: Figure 1 ﬁxes the training method – gradient (cid:96) 1 -regularization – and plots the obtained adversarial vulnerabilities for various attacks types.
Revised: Figure 1 ﬁxes the training method – gradient (cid:96) 1 -regularization – and plots the obtained adversarial vulnerabilities for various attacks types.

Original: Figure 2 ﬁxes the attack type – iterative (cid:96) ∞ -attacks – but plots the curves obtained for various training methods.
Revised: Figure 2 ﬁxes the attack type – iterative (cid:96) ∞ -attacks – but plots the curves obtained for various training methods.

Original: Note that our goal here is not to advocate one defense over another, but rather to check the validity of the Taylor expansion, and empirically verify that ﬁrst order terms (i.e., gradients) sufﬁce to explain much of the observed adversarial vulnerability.
Revised: Note that our goal here is not to advocate one defense over another, but rather to check the validity of the Taylor expansion, and empirically verify that ﬁrst order terms (i.e., gradients) sufﬁce to explain much of the observed adversarial vulnerability.

Original: Similarly, our goal in testing several attacks (Figure 1) is not to present a speciﬁcally strong one, but rather to verify that for all attacks, the trends are the same: the vulnerability grows with increasing gradients.
Revised: Similarly, our goal in testing several attacks (Figure 1) is not to present a speciﬁcally strong one, but rather to verify that for all attacks, the trends are the same: the vulnerability grows with increasing gradients.

Original: Validity of ﬁrst order expansion.
Revised: Validity of ﬁrst order expansion.

Original: The following observations support the validity of the ﬁrst order
Revised: The following observations support the validity of the ﬁrst order

Original: Taylor expansion in (2) and suggest that it is a crucial component of adversarial vulnerability: (i) the efﬁciency of the ﬁrst-order defense against iterative (non-ﬁrst-order) attacks (Fig.1a); (ii) the striking similarity between the PGD curves (adversarial augmentation with iterative attacks) and the other adversarial training training curves ( one-step attacks/defenses); (iii) the functional-like dependence between any approximation of adversarial vulnerability and E x (cid:107) ∂ x L(cid:107) 1 (Fig.1b), and its independence on the training method (Fig.2d).
Revised: Taylor expansion in (2) and suggest that it is a crucial component of adversarial vulnerability: (i) the efﬁciency of the ﬁrst-order defense against iterative (non-ﬁrst-order) attacks (Fig.1a); (ii) the striking similarity between the PGD curves (adversarial augmentation with iterative attacks) and the other adversarial training training curves ( one-step attacks/defenses); (iii) the functional-like dependence between any approximation of adversarial vulnerability and E x (cid:107) ∂ x L(cid:107) 1 (Fig.1b), and its independence on the training method (Fig.2d).

Original: (iv) the excellent correspondence between the gradient-regularization and adversarial training curves (see next paragraph).
Revised: (iv) the excellent correspondence between the gradient-regularization and adversarial training curves (see next paragraph).

Original: Said differently, adversarial examples seem indeed to be primarily caused by large gradients of the classiﬁer as captured via the induced loss.
Revised: Said differently, adversarial examples seem indeed to be primarily caused by large gradients of the classiﬁer as captured via the induced loss.

Original: Illustration of Proposition 3.
Revised: Illustration of Proposition 3.

Original: The upper row of Figure 2 plots E x (cid:107) ∂
Revised: The upper row of Figure 2 plots E x (cid:107) ∂

Original: x L 1 (cid:107) , adversarial vulnerability and accuracy as a function of (cid:15) d 1 /p .
Revised: x L 1 (cid:107) , adversarial vulnerability and accuracy as a function of (cid:15) d 1 /p .

Original: The excellent match between the adversarial augmentation curve with p = ∞ ( p = 2 ) and its gradient-regularization dual counterpart with q = 1 (resp.
Revised: The excellent match between the adversarial augmentation curve with p = ∞ ( p = 2 ) and its gradient-regularization dual counterpart with q = 1 (resp.

Original: q = 2 ) illustrates the duality between (cid:15) as a threshold for adversarially-augmented training and as a regularization constant in the regularized loss (Proposition 3).
Revised: q = 2 ) illustrates the duality between (cid:15) as a threshold for adversarially-augmented training and as a regularization constant in the regularized loss (Proposition 3).

Original: It also supports the validity of the ﬁrst-order Taylor expansion in (2) .
Revised: It also supports the validity of the ﬁrst-order Taylor expansion in (2) .

Original: Conﬁrmation of (3) .
Revised: Conﬁrmation of (3) .

Original: Still on the upper row, the curves for p = ∞ , q = 1 have no reason to match those for p = q = 2 when plotted against (cid:15) , because (cid:15) -threshold is relative to a speciﬁc attack-norm.
Revised: Still on the upper row, the curves for p = ∞ , q = 1 have no reason to match those for p = q = 2 when plotted against (cid:15) , because (cid:15) -threshold is relative to a speciﬁc attack-norm.

Original: However, (3) suggested that the rescaled thresholds (cid:15)d 1 /p may approximately correspond to a same ‘threshold-unit’ across (cid:96) p -norms and across dimension.
Revised: However, (3) suggested that the rescaled thresholds (cid:15)d 1 /p may approximately correspond to a same ‘threshold-unit’ across (cid:96) p -norms and across dimension.

Original: This is well conﬁrmed by the upper row plots: by rescaling the x-axis, the p = q = 2 and q = 1 , p = ∞ curves get almost super-imposed.
Revised: This is well conﬁrmed by the upper row plots: by rescaling the x-axis, the p = q = 2 and q = 1 , p = ∞ curves get almost super-imposed.

Original: Accuracy-vs-Vulnerability Trade-Off.
Revised: Accuracy-vs-Vulnerability Trade-Off.

Original: Merging Figures 2b and 2c by taking out (cid:15) , Figure 2f shows that all gradient regularization and adversarial training methods yield equivalent accuracyvulnerability trade-offs.
Revised: Merging Figures 2b and 2c by taking out (cid:15) , Figure 2f shows that all gradient regularization and adversarial training methods yield equivalent accuracyvulnerability trade-offs.

Original: Incidentally, for higher penalization values, these trade-offs appear to be much better than those given by cross Lipschitz regularization.
Revised: Incidentally, for higher penalization values, these trade-offs appear to be much better than those given by cross Lipschitz regularization.

Original: The penalty-norm does not matter.
Revised: The penalty-norm does not matter.

Original: We were surprised to see that on Figures 2d and 2f, the L (cid:15),q curves are almost identical for q = 1 and 2 .
Revised: We were surprised to see that on Figures 2d and 2f, the L (cid:15),q curves are almost identical for q = 1 and 2 .

Original: This indicates that both norms can be used interchangeably in (4) (modulo proper rescaling of (cid:15) via (3)), and suggests that protecting against a speciﬁc attacknorm also protects against others.
Revised: This indicates that both norms can be used interchangeably in (4) (modulo proper rescaling of (cid:15) via (3)), and suggests that protecting against a speciﬁc attacknorm also protects against others.

Original: (6) may provide an explanation: if the coordinates of ∂ x L behave like centered, uncorrelated variables with equal variance –which follows from assumptions ( H ) –, then the (cid:96) 1 - and (cid:96) 2 -norms of ∂ x L are simply proportional.
Revised: (6) may provide an explanation: if the coordinates of ∂ x L behave like centered, uncorrelated variables with equal variance –which follows from assumptions ( H ) –, then the (cid:96) 1 - and (cid:96) 2 -norms of ∂ x L are simply proportional.

Original: Plotting E x (cid:107) ∂
Revised: Plotting E x (cid:107) ∂

Original: x L ( x ) (cid:107) 2 against E x (cid:107) ∂
Revised: x L ( x ) (cid:107) 2 against E x (cid:107) ∂

Original: x L ( x ) (cid:107) 1 in Figure 2e conﬁrms this explanation.
Revised: x L ( x ) (cid:107) 1 in Figure 2e conﬁrms this explanation.

Original: The slope is independent of the training method.
Revised: The slope is independent of the training method.

Original: Therefore, penalizing (cid:107) ∂ x L ( x ) (cid:107) 1 during training will not only decrease E x (cid:107) ∂ x L(cid:107) 1 (as shown in Figure 2a), but also drive down E x (cid:107) ∂ x L(cid:107) 2 and vice-versa.
Revised: Therefore, penalizing (cid:107) ∂ x L ( x ) (cid:107) 1 during training will not only decrease E x (cid:107) ∂ x L(cid:107) 1 (as shown in Figure 2a), but also drive down E x (cid:107) ∂ x L(cid:107) 2 and vice-versa.

Original: 4. V ULNERABILITY G ROWS WITH I NPUT R ESOLUTION
Revised: 4. V ULNERABILITY G ROWS WITH I NPUT R ESOLUTION

Original: Theorems 4-5 and Corollary 6 predict a linear growth of the average (cid:96) 1 -norm of ∂ x L with the square root of the input dimension d , and therefore also of adversarial vulnerability (Lemma 2).
Revised: Theorems 4-5 and Corollary 6 predict a linear growth of the average (cid:96) 1 -norm of ∂ x L with the square root of the input dimension d , and therefore also of adversarial vulnerability (Lemma 2).

Original: To test these predictions, we created a 12 -class dataset of approximately 80 , 000 256 × 256 × 3 -sized RGB-images by merging similar ImageNet-classes, resizing the smallest image-edge to 256 pixels and center-cropping the result. We then downsized the images to 32 , 64, 128 and 256 pixels per edge, and trained 10 CNNs
Revised: To test these predictions, we upsampled the CIFAR-10 images (of size 3 x 32 x 32) by copying pixels so as to get 3 datasets with, respectively, 32, 64 and 128 pixels per edge. We then trained a CNN on each dataset

Original: [Figure]
Revised: [Figure]

Original: increase linearly with the square-root of the imageresolution d , as predicted by Corollary 6. Adversarial vulnerability gets slightly dampened at higher dimension, probably because the ﬁrst-order approximation made in (2) becomes less and less valid.
Revised: vulnerability ( increases similarly and computed their adversarial vulnerability (with iterative (cid:96) ∞ -attacks, threshold (cid:15) ∞ = .

Original: on each of these downsized datasets. We then computed adversarial vulnerability (with iterative (cid:96) ∞ -attacks) and average (cid:107) ∂ x L(cid:107) 1 for each network on a same held-out test-dataset. Figuresummarizes the results.
Revised: 002 ) and average (cid:107) ∂ x L(cid:107) 1 over the last 20 epochs on the same held-out test-dataset. This gave us 2 x 20-values per net and imagesize, summarized in Figure 3.

Original: The dashedline follows the median of each group of 10 networks; the errorbars show the 10 th and 90 th quantiles.
Revised: The dashed-lines follow their medians and the errorbars show their 10 th and 90 th quantiles.

Original: As predicted by our theorems, both (cid:107) ∂ x L(cid:107) 1 and adversarial vulnerability grow approximately linearly with √ d .
Revised: As predicted by our theorems, both (cid:107) ∂ x L(cid:107) 1 and adversarial vulnerability grow approximately linearly with √ d .

Original: As the gradients get much larger at higher dimensions, the ﬁrst order approximation in (2) becomes less and less valid, which explains the little inﬂection of the adversarial vulnerability curve. For smaller (cid:15) -thresholds, we veriﬁed that the inﬂection disappears.
Revised: We also ran a similar experiment on downsized ImageNet images, where we train several identical nets per image-size rather than just one. Conclusions are unchanged. See Appendix E.

Original: All networks had exactly the same amount of parameters and very similar structure across the various input-resolutions.
Revised: All networks had exactly the same amount of parameters and very similar structure across the various input-resolutions.

Original: The CNNs were a succession of 8 ‘convolution → batchnorm → ReLU’ layers with 64 output channels, followed by a ﬁnal full-connection to the 12 logit-outputs.
Revised: The CNNs were a succession of 8 ‘convolution → batchnorm → ReLU’ layers with 64 output channels, followed by a ﬁnal full-connection to the 12 logit-outputs.

Original: We used 2 × 2max-poolings after  layers 2,4 and 6, and a ﬁnal max-pooling after layer 8 that fed only 1 neuron per channel to the fully-connected layer.
Revised: We used 2 × 2max-poolings after the convolutions of layers 2,4, 6 and 8, and a ﬁnal max-pooling after layer 8 that fed only 1 neuron per channel to the fully-connected layer.

Original: To ensure that the convolution-kernels cover similar ranges of the images across each of the 32, 64, 128 and 256 input-resolutions, we respectively dilated all convolutions (‘à trous’) by a factor 1 , 2 , 4 and 8 . 
Revised: To ensure that the convolution-kernels cover similar ranges of the images across each of the 32, 64, 128 and 256 input-resolutions, we respectively dilated all convolutions (‘à trous’) by a factor 1 , 2 , 4 and 8 . [Figure]

Revised:  D ISCUSSIONS 5. I MPLICATIONS : WHY PRIOR VULNERABILITY MAY MATTER

Revised: Our theoretical results show that the priors of classical neural networks yield vulnerable functions because of naturally high gradients.

Revised: And our experiments (Fig 3&6) suggest that usual training does not escape these prior properties.

Revised: But how may these insights help understanding the vulnerability of robustly trained networks?

Original: R ELATED L ITERATURE
Revised: Clearly, to be successful, robust training algorithms must escape ill-behaved priors, which explains why most methods (e.g. FGSM, PGD) are essentially gradient penalization techniques. But, MNIST aside, even state-of-the-art methods largely fail at protecting current network architectures (Madry et al., 2018), and understanding why is motivation to this and many other papers. Interestingly, Schmidt et al. (2018) recently noticed that those methods actually do protect the nets on training examples, but fail to generalize to the test set.

Revised: They hence conclude that state-of-the-art robustiﬁcation algorithms work, but need more data.

Revised: Alternatively however, when generalization fails, one can also reduce the model’s complexity.

Revised: Large fully connected nets for example typically fail to generalize to out-of-sample examples: getting similar accuracies than CNNs would need prohibitively many training points.

Revised: Similarly, Schmidt et al.

Revised: ’s

Revised: observations may suggest that, outside the training points, networks tend to recover their prior properties, i.e. naturally large gradients.

Revised: Figure 4 corroborates this hypothesis.

Revised: It plots the evolution over training epochs of the (cid:96) 1 -gradient-norms of the CNNs from Section 4.2 (Fig 3) on the training and test sets respectively.

Revised: The discrepancy is unmistakable: after a brief initialization phase, the norms decrease on the training set, but increase on the test set.

Revised: They are moreover almost input-dimension independent on the training set, but scale as √ d on the test set (as seen in Fig 4.2) up to respectively 2, 4 and 8 times the training set values.

Revised: These observations suggest that, with the current amount of data, tackling adversarial vulnerability may require new architectures with inherently smaller gradients.

Revised: Searching these architectures among those with well-behaved prior-gradients seems a reasonable start, where our theoretical results may prove very useful.

Revised: 5. R ELATED L ITERATURE

Revised: On network vulnerability.

Original: Goodfellow et al. (2015) already stressed that adversarial vulnerability increases with growing dimension d .
Revised: Goodfellow et al. (2015) already stressed that adversarial vulnerability increases with growing dimension d .

Original: But their argument only relied on a linear ‘one-output-to-many-inputs’-model with dimension-independent weights.
Revised: But their argument only relied on a linear ‘one-output-to-manyinputs’-model with dimension-independent weights.

Original: They therefore concluded on a linear growth of adversarial vulnerability with d .
Revised: They therefore concluded on a linear growth of adversarial vulnerability with d .

Original: In contrast, our theory applies to almost any standard feed-forward architecture (not just linear), and shows that, once we adjust for the weight’s dimension-dependence, adversarial vulnerability increases like √ d (not d ), almost independently of the architecture .
Revised: In contrast, our theory applies to almost any standard feed-forward architecture (not just linear), and shows that, once we adjust for the weight’s dimension-dependence, adversarial vulnerability increases like √ d (not d ), almost independently of the architecture .

Original: Nevertheless, our experiments conﬁrm Goodfellow et al.
Revised: Nevertheless, our experiments conﬁrm Goodfellow et al.

Original: ’s idea that our networks are “too linear-like”, in the sense that a ﬁrst-order Taylor expansion is indeed sufﬁcient to explain the adversarial vulnerability of neural networks.
Revised: ’s idea that our networks are “too linear-like”, in the sense that a ﬁrst-order Taylor expansion is indeed sufﬁcient to explain the adversarial vulnerability of neural networks.

Original: As suggested by the one-output-to-many-inputs model, the culprit is that growing dimensionality gives the adversary more and more room to ‘wriggle around’ with the noise and adjust to the gradient of the output neuron.
Revised: As suggested by the one-output-to-many-inputs model, the culprit is that growing dimensionality gives the adversary more and more room to ‘wriggle around’ with the noise and adjust to the gradient of the output neuron.

Original: This wriggling, we show, is still possible when the output is connected to all inputs only indirectly, even when no neuron is directly connected to all inputs, like in
Revised: This wriggling, we show, is still possible when the output is connected to all inputs only indirectly, even when no neuron is directly connected to all inputs, like in

Original: CNNs.
Revised: CNNs.

Original: This explanation of adversarial vulnerability is independent of the intrinsic dimensionality or geometry of the data (compare to Amsaleg et al. 2017;
Revised: This explanation of adversarial vulnerability is independent of the intrinsic dimensionality or geometry of the data (compare to Amsaleg et al. 2017;

Original: Gilmer et al. 2018).
Revised: Gilmer et al. 2018).

Revised: Finally, let us mention that Fawzi et al.

Revised: (2016) show a close link between the vulnerability to small worst-case perturbation (as studied here) and larger average perturbations.

Revised: Our ﬁndings on the adversarial vulnerability NNs to small perturbation could thus be translated accordingly.

Original:  Incidentally, Goodfellow et al. (2015) also already relate adversarial vulnerability to large gradients of the loss L , an insight at the very heart of their FGSM-algorithm.
Revised: On robustiﬁcation algorithms. Incidentally, Goodfellow et al. (2015) also already relate adversarial vulnerability to large gradients of the loss L , an insight at the very heart of their FGSM-algorithm.

Original: They however do not propose any explicit penalizer on the gradient of L other than indirectly through adversarially-augmented training.
Revised: They however do not propose any explicit penalizer on the gradient of L other than indirectly through adversarially-augmented training.

Original: Conversely, Ross & Doshi-Velez (2018) propose the old double-backpropagation to robustify networks but make no connection to FGSM and adversarial augmentation.
Revised: Conversely, Ross & Doshi-Velez (2018) propose the old double-backpropagation to robustify networks but make no connection to FGSM and adversarial augmentation.

Original: Lyu et al.
Revised: Lyu et al.

Original: (2015) discuss and use the connection between gradient-penalties and adversarial augmentation, but never actually compare both in experiments.
Revised: (2015) discuss and use the connection between gradient-penalties and adversarial augmentation, but never actually compare both in experiments.

Original: This comparison however is essential to test the validity of the ﬁrstorder Taylor expansion in (2), as conﬁrmed by the similarity between the gradient-regularization and adversarial-augmentation curves in Figure 2.
Revised: This comparison however is essential to test the validity of the ﬁrst-order Taylor expansion in (2), as conﬁrmed by the similarity between the gradient-regularization and adversarial-augmentation curves in Figure 2.

Original: Hein & Andriushchenko (2017) derived yet another gradient-based penalty –the cross-Lipschitz -penalty– by considering (and proving) formal guarantees on adversarial vulnerability itself, rather than adversarial damage.
Revised: Hein & Andriushchenko (2017) derived yet another gradient-based penalty –the cross-Lipschitz -penalty– by considering (and proving) formal guarantees on adversarial vulnerability itself, rather than adversarial damage.

Original: While both penalties are similar in spirit, focusing on the adversarial damage rather than vulnerability has two main advantages.
Revised: While both penalties are similar in spirit, focusing on the adversarial damage rather than vulnerability has two main advantages.

Original: First, it achieves better accuracy-to-vulnerability ratios, both in theory and practice, because it ignores classswitches between misclassiﬁed examples and penalizes only those that reduce the accuracy.
Revised: First, it achieves better accuracy-to-vulnerability ratios, both in theory and practice, because it ignores class-switches between misclassiﬁed examples and penalizes only those that reduce the accuracy.

Original: Second, it allows to deal with one number only, ∆ L , whereas Hein & Andriushchenko’s cross-Lipschitz regularizer and theoretical guarantees explicitly involve all K logit-functions (and their gradients).
Revised: Second, it allows to deal with one number only, ∆ L , whereas Hein & Andriushchenko’s cross-Lipschitz regularizer and theoretical guarantees explicitly involve all K logit-functions (and their gradients).

Original: See Appendix C. Penalizing network-gradients is also at the heart of contractive auto-encoders as proposed by Rifai et al.
Revised: See Appendix C. Penalizing network-gradients is also at the heart of contractive auto-encoders as proposed by Rifai et al.

Original: (2011), where it is used to regularize the encoder-features.
Revised: (2011), where it is used to regularize the encoder-features.

Original: Seeing adversarial training as a generalization method, let us also mention Hochreiter & Schmidhuber (1995), who propose to enhance generalization by searching for parameters in a “ﬂat minimum region” of the loss.
Revised: Seeing adversarial training as a generalization method, let us also mention Hochreiter & Schmidhuber (1995), who propose to enhance generalization by searching for parameters in a “ﬂat minimum region” of the loss.

Original: This leads to a penalty involving the gradient of the loss, but taken with respect to the weights, rather than the inputs.
Revised: This leads to a penalty involving the gradient of the loss, but taken with respect to the weights, rather than the inputs.

Original: In the same vein, a gradient-regularization of the loss of generative models also appears in Proposition 6 of Ollivier (2014), where it stems from a code-length bound on the data (minimum description length).
Revised: In the same vein, a gradientregularization of the loss of generative models also appears in Proposition 6 of Ollivier (2014), where it stems from a code-length bound on the data (minimum description length).

Original: More generally, the gradient regularized objective (4) is essentially the ﬁrst-order approximation of the robust training objective max (cid:107) δ (cid:107)≤ (
Revised: More generally, the gradient regularized objective (4) is essentially the ﬁrst-order approximation of the robust training objective max (cid:107) δ (cid:107)≤ (

Original: cid:15) L ( x + δ , c ) which has a long history in math (Wald, 1945), machine learning (Xu et al., 2009) and now adversarial vulnerability (Sinha et al., 2018).
Revised: cid:15) L ( x + δ , c ) which has a long history in math (Wald, 1945), machine learning (Xu et al., 2009) and now adversarial vulnerability (Sinha et al., 2018).

Original: Finally, Cisse et al.
Revised: Finally, Cisse et al.

Original: (2017) propose new network-architectures that have small gradients by design, rather than by special training: an approach that makes all the more sense, considering the conclusion of Theorems 4 and 5.
Revised: (2017) propose new network-architectures that have small gradients by design, rather than by special training: an approach that makes all the more sense, considering the conclusion of Theorems 4 and 5.

Original: For further details and references on adversarial attacks and defenses, we refer to Yuan et al.
Revised: For further details and references on adversarial attacks and defenses, we refer to Yuan et al.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: For differentiable classiﬁers and losses, we showed that adversarial vulnerability increases with the gradients ∂ x L of the loss, which is conﬁrmed by the near-perfect functional relationship between gradient norms and vulnerability (Figures 1&2d).
Revised: For differentiable classiﬁers and losses, we showed that adversarial vulnerability increases with the gradients ∂ x L of the loss, which is conﬁrmed by the near-perfect functional relationship between gradient norms and vulnerability (Figures 1&2d).

Original: We then evaluated the size of (cid:107) ∂ x L(cid:107) q and showed that usual feed-forward nets (convolutional or fully connected) are increasingly vulnerable to (cid:96) pattacks with growing input dimension d (the image-size), almost independently of their architecture.
Revised: We then evaluated the size of (cid:107) ∂ x L(cid:107) q and showed that, at initialization, usual feed-forward nets (convolutional or fully connected) are increasingly vulnerable to (cid:96) p -attacks with growing input dimension d (the image-size), almost independently of their architecture.

Original: Our theorems rely on the statistical weight distribution at initialization, but our experiments conﬁrm the conclusions also for the tested networks after training.
Revised: Our experiments show that, on the tested architectures, usual training escapes those prior gradient (and vulnerability) properties on the training, but not on the test set.

Original: Our results rely on a ﬁrst order analysis that assumes a differentiable loss and architecture: they may not cover every aspect of adversarial vulnerability nor easily extend to non-differentiable structures (even though such structures are often smooth at a coarser scale).
Revised: Schmidt et al.

Original: Nevertheless, they show that at least this type of ﬁrst-order vulnerability is present, common, andﬁrmly rooted in  our current network architectures. They hence suggest to tackle adversarial vulnerability by designing new architectures (or new architectural building blocks) rather than by new regularization techniques
Revised: (2018) suggest that alleviating this generalization gap requires more data. But a natural (complementary) alternative would be to search for architectures with naturally smaller gradients, and in particular, with well-behaved priors. Despite all their limitations (being only ﬁrst-order, assuming a prior weight-distribution and a differentiable loss and architecture), our theoretical insights may thereby still prove to be precious future allies.

Original: A E FFECTS OF S TRIDED AND A VERAGE -P OOLING L AYERS ON A DVERSARIAL V ULNERABILITY
Revised: A E FFECTS OF S TRIDED AND A VERAGE -P OOLING L AYERS ON A DVERSARIAL V ULNERABILITY

Original: It is common practice in CNNs to use average-pooling layers or strided convolutions to progressively decrease the number of pixels per channel.
Revised: It is common practice in CNNs to use average-pooling layers or strided convolutions to progressively decrease the number of pixels per channel.

Original: Corollary 6 shows that using strided convolutions does not protect against adversarial examples.
Revised: Corollary 6 shows that using strided convolutions does not protect against adversarial examples.

Original: However, what if we replace strided convolutions by convolutions with stride 1 plus an average-pooling layer?
Revised: However, what if we replace strided convolutions by convolutions with stride 1 plus an average-pooling layer?

Original: Theorem 5 considers only randomly initialized weights with typical size 1 / √ in-degree.
Revised: Theorem 5 considers only randomly initialized weights with typical size 1 / √ in-degree.

Original: Average-poolings however introduce deterministic weights of size 1 / ( in-degree ) .
Revised: Average-poolings however introduce deterministic weights of size 1 / ( in-degree ) .

Original: These are smaller and may therefore dampen the input-to-output gradients and protect against adversarial examples.
Revised: These are smaller and may therefore dampen the input-to-output gradients and protect against adversarial examples.

Original: We conﬁrm this in our next theorem, which uses a slightly modiﬁed version ( H (cid:48) ) of ( H ) to allow average pooling layers.
Revised: We conﬁrm this in our next theorem, which uses a slightly modiﬁed version ( H (cid:48) ) of ( H ) to allow average pooling layers.

Original: ( H (cid:48) ) is ( H ), but where the
Revised: ( H (cid:48) ) is ( H ), but where the

Original: He-init H3 applies to all weights except the (deterministic) average pooling weights, and where H places a ReLU on every non-input and non-average-pooling neuron.
Revised: He-init H3 applies to all weights except the (deterministic) average pooling weights, and where H places a ReLU on every non-input and non-average-pooling neuron.

Original: Theorem 7 ( Effect of Average-Poolings ) .
Revised: Theorem 7 ( Effect of Average-Poolings ) .

Original: Consider a succession of convolution layers, dense layers and n average-pooling layers, in any order, that satisﬁes ( H (cid:48) ) and outputs logits f k ( x ) .
Revised: Consider a succession of convolution layers, dense layers and n average-pooling layers, in any order, that satisﬁes ( H (cid:48) ) and outputs logits f k ( x ) .

Original: Assume the n average pooling layers have a stride equal to their mask size and perform averages over a 1 , ..., a n nodes respectively.
Revised: Assume the n average pooling layers have a stride equal to their mask size and perform averages over a 1 , ..., a n nodes respectively.

Original: Then (cid:107) ∂
Revised: Then (cid:107) ∂

Original: x f k (cid:107) 2 and | ∂ x f k | scale like 1 / √ a 1 · · · a n and 1 / √ d
Revised: x f k (cid:107) 2 and | ∂ x f k | scale like 1 / √ a 1 · · · a n and 1 / √ d

Original: a 1 · · · a n respectively.
Revised: a 1 · · · a n respectively.

Original: Proof in Appendix B.4.
Revised: Proof in Appendix B.4.

Original: Theorem 7 suggest to try and replace any strided convolution by its non-strided counterpart, followed by an average-pooling layer.
Revised: Theorem 7 suggest to try and replace any strided convolution by its non-strided counterpart, followed by an average-pooling layer.

Original: It also shows that if we systematically reduce the number of pixels per channel down to 1 by using only non-strided convolutions and average-pooling layers (i.e. d = (cid:81) ni =1 a i ), then all input-to-output gradients should become independent of d , thereby making the network completely robust to adversarial examples.
Revised: It also shows that if we systematically reduce the number of pixels per channel down to 1 by using only non-strided convolutions and average-pooling layers (i.e. d = (cid:81) ni =1 a i ), then all input-to-output gradients should become independent of d , thereby making the network completely robust to adversarial examples.

Original: [Figure]
Revised: [Figure]

Original: Our following experiments (Figure 4) show that after training, the networks get indeed robustiﬁed to adversarial examples, but remain more vulnerable than suggested by Theorem 7.
Revised: Our following experiments (Figure 5) show that after training, the networks get indeed robustiﬁed to adversarial examples, but remain more vulnerable than suggested by Theorem 7.

Original: Experimental setup.
Revised: Experimental setup.

Original: Theorem 7 shows that, contrary to strided layers, average-poolings should decrease adversarial vulnerability.
Revised: Theorem 7 shows that, contrary to strided layers, average-poolings should decrease adversarial vulnerability.

Original: We tested this hypothesis on CNNs trained on
Revised: We tested this hypothesis on CNNs trained on

Original: CIFAR-10, with 6 blocks of ‘convolution →
Revised: CIFAR-10, with 6 blocks of ‘convolution →

Original: BatchNorm → ReLU’ with 64 output-channels, followed by a ﬁnal average pooling feeding one neuron per channel to the last fully-connected linear layer.
Revised: BatchNorm → ReLU’ with 64 output-channels, followed by a ﬁnal average pooling feeding one neuron per channel to the last fully-connected linear layer.

Original: Additionally, after every second convolution, we placed a pooling layer with stride and mask-size (2 , 2) (thus acting on 2 ×neurons at a time, without overlap).
Revised: Additionally, after every second convolution, we placed a pooling layer with stride and mask-size (2 , 2) (thus acting on 2 ×neurons at a time, without overlap).

Original: We tested average-pooling, strided and max-pooling layers and trained 20 networks per architecture.
Revised: We tested average-pooling, strided and max-pooling layers and trained 20 networks per architecture.

Original: Results are shown in Figure 4.
Revised: Results are shown in Figure 5.

Original: All accuracies are very close, but, as predicted, the networks with average pooling layers are more robust to adversarial images than the others.
Revised: All accuracies are very close, but, as predicted, the networks with average pooling layers are more robust to adversarial images than the others.

Original: However, they remain more vulnerable than what would follow from Theorem 7.
Revised: However, they remain more vulnerable than what would follow from Theorem 7.

Original: We also noticed that, contrary to the strided architectures, their gradients after training are an order of magnitude higher than at initialization and than predicted.
Revised: We also noticed that, contrary to the strided architectures, their gradients after training are an order of magnitude higher than at initialization and than predicted.

Original: This suggests that assumptions ( H ) get more violated when using average-poolings instead of strided layers.
Revised: This suggests that assumptions ( H ) get more violated when using average-poolings instead of strided layers.

Original: Understanding why will need further investigations.
Revised: Understanding why will need further investigations.

Original: B P ROOFS B. P ROOF OF P ROPOSITIONProof
Revised: B P ROOFS B. P ROOF OF P ROPOSITIONProof

Original: Let (cid:15) δ be an adversarial perturbation with (cid:107) δ (cid:107) = 1 that locally maximizes the loss increase at point x , meaning that δ = arg max (cid:107) δ (cid:48) (cid:107)≤ 1 ∂ x L · δ (cid:48) .
Revised: Let (cid:15) δ be an adversarial perturbation with (cid:107) δ (cid:107) = 1 that locally maximizes the loss increase at point x , meaning that δ = arg max (cid:107) δ (cid:48) (cid:107)≤ 1 ∂ x L · δ (cid:48) .

Original: Then, by deﬁnition of the dual norm of ∂ x L we have: ∂ x L · ( (cid:15) δ ) =
Revised: Then, by deﬁnition of the dual norm of ∂ x L we have: ∂ x L · ( (cid:15) δ ) =

Original: (cid:15) |||
Revised: (cid:15) |||

Original: ∂ x L||| .
Revised: ∂ x L||| .

Original: Thus
Revised: Thus

Original: ˜
Revised: ˜

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Let x designate a generic coordinate of x .
Revised: Let x designate a generic coordinate of x .

Original: To evaluate the size of (cid:107) ∂ x L(cid:107) q , we will evaluate the size of the coordinates ∂ x L of ∂ x L by decomposing them into
Revised: To evaluate the size of (cid:107) ∂ x L(cid:107) q , we will evaluate the size of the coordinates ∂ x L of ∂ x L by decomposing them into

Original: [Equation]
Revised: [Equation]

Original: where f k ( x ) denotes the logit-probability of x belonging to class k .
Revised: where f k ( x ) denotes the logit-probability of x belonging to class k .

Original: We now investigate the statistical properties of the logit gradients ∂
Revised: We now investigate the statistical properties of the logit gradients ∂

Original: x f k , and then see how they shape ∂ x L .
Revised: x f k , and then see how they shape ∂ x L .

Original: Step 1: Statistical properties of ∂ x f k .
Revised: Step 1: Statistical properties of ∂ x f k .

Original: Let P ( x, k ) be the set of paths p from input neuron x to output-logit k .
Revised: Let P ( x, k ) be the set of paths p from input neuron x to output-logit k .

Original: Let p − 1 and p be two successive neurons on path p , and ˜ p be the same path p but without its input neuron.
Revised: Let p − 1 and p be two successive neurons on path p , and ˜ p be the same path p but without its input neuron.

Original: Let w p designate the weight from p − 1 to p and ω p be the path-product ω p :=
Revised: Let w p designate the weight from p − 1 to p and ω p be the path-product ω p :=

Original: (cid:81) p ∈ ˜ p w p .
Revised: (cid:81) p ∈ ˜ p w p .

Original: Finally, let σ p (resp.
Revised: Finally, let σ p (resp.

Original: σ p ) be equal to 1 if the ReLU of node p (resp.
Revised: σ p ) be equal to 1 if the ReLU of node p (resp.

Original: if path p ) is active for input x , and 0 otherwise.
Revised: if path p ) is active for input x , and 0 otherwise.

Original: As previously noticed by Balduzzi et al.
Revised: As previously noticed by Balduzzi et al.

Original: (2017) using the chain rule, we see that ∂ x f k is the sum of all ω p whose path is active, i.e. ∂ x f k ( x ) = (cid:80) p ∈P ( x,k ) ω p σ p .
Revised: (2017) using the chain rule, we see that ∂ x f k is the sum of all ω p whose path is active, i.e. ∂ x f k ( x ) = (cid:80) p ∈P ( x,k ) ω p σ p .

Original: Consequently:
Revised: Consequently:

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst equality uses H1 to decouple the expectations over weights and ReLUs, and then applies
Revised: The ﬁrst equality uses H1 to decouple the expectations over weights and ReLUs, and then applies

Original: Lemma 10 of Appendix B.3, which uses H3-H5 to kill all cross-terms and take the expectation over weights inside the product.
Revised: Lemma 10 of Appendix B.3, which uses H3-H5 to kill all cross-terms and take the expectation over weights inside the product.

Original: The second equality uses H3 and the fact that the resulting product is the same for all active paths.
Revised: The second equality uses H3 and the fact that the resulting product is the same for all active paths.

Original: The third equality counts the number of paths from x to k
Revised: The third equality counts the number of paths from x to k

Original: and we conclude by noting that all terms cancel out, except d p − 1 from the input layer which is d .
Revised: and we conclude by noting that all terms cancel out, except d p − 1 from the input layer which is d .

Original: Equation 8 shows that | ∂ x f k | ∝ 1 / √ d .
Revised: Equation 8 shows that | ∂ x f k | ∝ 1 / √ d .

Original: Step 2: Statistical properties of ∂ k L and ∂ x L .
Revised: Step 2: Statistical properties of ∂ k L and ∂ x L .

Original: Deﬁning q k
Revised: Deﬁning q k

Original: ( x ) :=
Revised: ( x ) :=

Original: [Equation]
Revised: [Equation]

Original: (the probability of image x belonging to class k according to the network), we have, by deﬁnition of the cross-entropy loss, L ( x , c ) :
Revised: (the probability of image x belonging to class k according to the network), we have, by deﬁnition of the cross-entropy loss, L ( x , c ) :

Original: = − log q c ( x ) , where c is the label of the target class.
Revised: = − log q c ( x ) , where c is the label of the target class.

Original: Thus:
Revised: Thus:

Original: [Equation]
Revised: [Equation]

Original: otherwise , and
Revised: otherwise , and

Original: [Equation]
Revised: [Equation]

Original: Using again Lemma 10, we see that the ∂ x f k ( x ) are K centered and uncorrelated variables.
Revised: Using again Lemma 10, we see that the ∂ x f k ( x ) are K centered and uncorrelated variables.

Original: So ∂ x L ( x ) is approximately the sum of K uncorrelated variables with zero-mean, and its total variance is given by (cid:0)
Revised: So ∂ x L ( x ) is approximately the sum of K uncorrelated variables with zero-mean, and its total variance is given by (cid:0)

Original: (1 − q c ) 2 + (cid:80) k
Revised: (1 − q c ) 2 + (cid:80) k

Original: (cid:54) =
Revised: (cid:54) =

Original: c q 2 k (cid:1) /d .
Revised: c q 2 k (cid:1) /d .

Original: Hence the magnitude of ∂ x L ( x ) is 1 / √ d for all x , so the (cid:96) q -norm of the full input gradient is d 1 /q − 1 / 2 .
Revised: Hence the magnitude of ∂ x L ( x ) is 1 / √ d for all x , so the (cid:96) q -norm of the full input gradient is d 1 /q − 1 / 2 .

Original: (6) concludes.
Revised: (6) concludes.

Original: Remark 1 .
Revised: Remark 1 .

Original: Equation 9 can be rewritten as
Revised: Equation 9 can be rewritten as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: As the term k = c disappears, the norm of the gradients ∂ x L ( x ) appears to be controlled by the total error probability.
Revised: As the term k = c disappears, the norm of the gradients ∂ x L ( x ) appears to be controlled by the total error probability.

Original: This suggests that, even without regularization, trying to decrease the ordinary classiﬁcation error is still a valid strategy against adversarial examples.
Revised: This suggests that, even without regularization, trying to decrease the ordinary classiﬁcation error is still a valid strategy against adversarial examples.

Original: It reﬂects the fact that when increasing the classiﬁcation margin, larger gradients of the classiﬁer’s logits are needed to push images from one side of the classiﬁcation boundary to the other.
Revised: It reﬂects the fact that when increasing the classiﬁcation margin, larger gradients of the classiﬁer’s logits are needed to push images from one side of the classiﬁcation boundary to the other.

Original: This is conﬁrmed by Theorem 2.1 of
Revised: This is conﬁrmed by Theorem 2.1 of

Original: Hein & Andriushchenko (2017).
Revised: Hein & Andriushchenko (2017).

Original: See also (16) in Appendix C.
Revised: See also (16) in Appendix C.

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: The proof of Theorem 5 is very similar to the one of Theorem 4, but we will need to ﬁrst generalize the equalities appearing in (8).
Revised: The proof of Theorem 5 is very similar to the one of Theorem 4, but we will need to ﬁrst generalize the equalities appearing in (8).

Original: To do so, we identify the computational graph of a neural network to an abstract Directed Acyclic Graph (DAG) which we use to prove the needed algebraic equalities.
Revised: To do so, we identify the computational graph of a neural network to an abstract Directed Acyclic Graph (DAG) which we use to prove the needed algebraic equalities.

Original: We then concentrate on the statistical weight-interactions implied by assumption ( H ), and ﬁnally throw these results together to prove the theorem.
Revised: We then concentrate on the statistical weight-interactions implied by assumption ( H ), and ﬁnally throw these results together to prove the theorem.

Original: In all the proof, o will designate one of the output-logits f k ( x ) .
Revised: In all the proof, o will designate one of the output-logits f k ( x ) .

Original: Lemma 8.
Revised: Lemma 8.

Original: Let x be the vector of inputs to a given DAG, o be any leaf-node of the DAG, x a generic coordinate of x .
Revised: Let x be the vector of inputs to a given DAG, o be any leaf-node of the DAG, x a generic coordinate of x .

Original: Let p be a path from the set of paths P ( x, o ) from x to o , ˜ p
Revised: Let p be a path from the set of paths P ( x, o ) from x to o , ˜ p

Original: the same path without node x , p a generic node in ˜ p , and d p be its input-degree.
Revised: the same path without node x , p a generic node in ˜ p , and d p be its input-degree.

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We will reason on a random walk starting at o and going up the DAG by choosing any incoming node with equal probability.
Revised: We will reason on a random walk starting at o and going up the DAG by choosing any incoming node with equal probability.

Original: The DAG being ﬁnite, this walk will end up at an input-node x with probability 1 .
Revised: The DAG being ﬁnite, this walk will end up at an input-node x with probability 1 .

Original: Each path p is taken with probability (cid:81)
Revised: Each path p is taken with probability (cid:81)

Original: p ∈ ˜
Revised: p ∈ ˜

Original: p 1 d p .
Revised: p 1 d p .

Original: And the probability to end up at an input-node is the sum of all these probabilities, i.e. (cid:80) x ∈ x (cid:80) p ∈P
Revised: And the probability to end up at an input-node is the sum of all these probabilities, i.e. (cid:80) x ∈ x (cid:80) p ∈P

Original: ( x,o ) (cid:81)
Revised: ( x,o ) (cid:81)

Original: p ∈ p
Revised: p ∈ p

Original: d − 1 p , which concludes.
Revised: d − 1 p , which concludes.

Original: The sum over all inputs x in (11) being 1, on average it is 1 /d for each x , where d is the total number of inputs (i.e. the length of x ).
Revised: The sum over all inputs x in (11) being 1, on average it is 1 /d for each x , where d is the total number of inputs (i.e. the length of x ).

Original: It becomes an equality under assumption ( S ):
Revised: It becomes an equality under assumption ( S ):

Original: Lemma 9.
Revised: Lemma 9.

Original: Under the symmetry assumption ( S ), and with the previous notations, for any input x ∈ x :
Revised: Under the symmetry assumption ( S ), and with the previous notations, for any input x ∈ x :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let us denote D ( x, o ) :
Revised: Let us denote D ( x, o ) :

Original: = { d p } x ∈P ( x,o ) .
Revised: = { d p } x ∈P ( x,o ) .

Original: Each path p in P ( x, o ) corresponds to exactly one element d p in D ( x, o ) and vice-versa.
Revised: Each path p in P ( x, o ) corresponds to exactly one element d p in D ( x, o ) and vice-versa.

Original: And the elements d p of d p completely determine the product (cid:81)
Revised: And the elements d p of d p completely determine the product (cid:81)

Original: p ∈ ˜ p
Revised: p ∈ ˜ p

Original: d − 1 p .
Revised: d − 1 p .

Original: By using (11) and the fact that, by ( S ), the multiset D ( x, o ) is independent of x , we hence conclude
Revised: By using (11) and the fact that, by ( S ), the multiset D ( x, o ) is independent of x , we hence conclude

Original: [Equation]
Revised: [Equation]

Original: Now, let us relate these considerations on graphs to gradients and use assumptions ( H ).
Revised: Now, let us relate these considerations on graphs to gradients and use assumptions ( H ).

Original: We remind that path-product ω p is the product (cid:81)
Revised: We remind that path-product ω p is the product (cid:81)

Original: p ∈ ˜
Revised: p ∈ ˜

Original: p w p .
Revised: p w p .

Original: Lemma 10.
Revised: Lemma 10.

Original: Under assumptions ( H ), the path-products ω p , ω p (cid:48) of two distinct paths p and p (cid:48) starting from a same input node x , satisfy:
Revised: Under assumptions ( H ), the path-products ω p , ω p (cid:48) of two distinct paths p and p (cid:48) starting from a same input node x , satisfy:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, if there is at least one non-average-pooling weight on path p , then E W [ ω p ] = 0 .
Revised: Furthermore, if there is at least one non-average-pooling weight on path p , then E W [ ω p ] = 0 .

Original: Proof.
Revised: Proof.

Original: Hypothesis H4 yields
Revised: Hypothesis H4 yields

Original: [Equation]
Revised: [Equation]

Original: Now, take two different paths p and p (cid:48) that start at a same node x .
Revised: Now, take two different paths p and p (cid:48) that start at a same node x .

Original: Starting from x , consider the ﬁrst node after which p and p (cid:48) part and call p and p (cid:48)
Revised: Starting from x , consider the ﬁrst node after which p and p (cid:48) part and call p and p (cid:48)

Original: the next nodes on p and p (cid:48) respectively.
Revised: the next nodes on p and p (cid:48) respectively.

Original: Then the weights w p and w p (cid:48) are two weights of a same node.
Revised: Then the weights w p and w p (cid:48) are two weights of a same node.

Original: Applying H4 and H5 hence gives
Revised: Applying H4 and H5 hence gives

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: if p has at least one non-average-pooling node p
Revised: if p has at least one non-average-pooling node p

Original: [Equation]
Revised: [Equation]

Original: then successively applying H4 and H yields:
Revised: then successively applying H4 and H yields:

Original: [Equation]
Revised: [Equation]

Original: We now have all elements to prove Theorem 5.
Revised: We now have all elements to prove Theorem 5.

Original: Proof.
Revised: Proof.

Original: ( of Theorem 5 )
Revised: ( of Theorem 5 )

Original: For a given neuron p in ˜ p
Revised: For a given neuron p in ˜ p

Original: [Equation]
Revised: [Equation]

Original: let p − 1 designate the previous node in p of p .
Revised: let p − 1 designate the previous node in p of p .

Original: Let σ p (resp.
Revised: Let σ p (resp.

Original: σ p ) be a variable equal to 0 if neuron p gets killed by its ReLU (resp.
Revised: σ p ) be a variable equal to 0 if neuron p gets killed by its ReLU (resp.

Original: path p is inactive), and 1 otherwise.
Revised: path p is inactive), and 1 otherwise.

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: Consequently:
Revised: Consequently:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrs line uses the independence between the ReLU killings and the weights (H1), the second uses Lemma 10 and the last uses Lemma 9.
Revised: where the ﬁrs line uses the independence between the ReLU killings and the weights (H1), the second uses Lemma 10 and the last uses Lemma 9.

Original: The gradient ∂ x o thus has coordinates whose squared expectations scale like 1 /d .
Revised: The gradient ∂ x o thus has coordinates whose squared expectations scale like 1 /d .

Original: Thus each coordinate scales like 1 / √ d and (cid:107) ∂
Revised: Thus each coordinate scales like 1 / √ d and (cid:107) ∂

Original: x o (cid:107) q like d 1 / 2 − 1 /q .
Revised: x o (cid:107) q like d 1 / 2 − 1 /q .

Original: Conclude on (cid:107) ∂ x L(cid:107) q
Revised: Conclude on (cid:107) ∂ x L(cid:107) q

Original: and (cid:15) p (cid:107) ∂ x L(cid:107) q by using Step 2 of the proof of Theorem 4.
Revised: and (cid:15) p (cid:107) ∂ x L(cid:107) q by using Step 2 of the proof of Theorem 4.

Original: Finally, note that, even without the symmetry assumption ( S ), using Lemma 8 shows that
Revised: Finally, note that, even without the symmetry assumption ( S ), using Lemma 8 shows that

Original: [Equation]
Revised: [Equation]

Original: Thus, with or without ( S ), (cid:107) ∂ x o (cid:107) 2 is independent of the input-dimension d .
Revised: Thus, with or without ( S ), (cid:107) ∂ x o (cid:107) 2 is independent of the input-dimension d .

Original: B. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: To prove Theorem 7, we will actually prove the following more general theorem, which generalizes
Revised: To prove Theorem 7, we will actually prove the following more general theorem, which generalizes

Original: Theorem 5.
Revised: Theorem 5.

Original: Theorem 7 is a straightforward corollary of it.
Revised: Theorem 7 is a straightforward corollary of it.

Original: Theorem 11.
Revised: Theorem 11.

Original: Consider any feed-forward network with linear connections and ReLU activation functions that outputs logits f k ( x ) and satisﬁes assumptions ( H ).
Revised: Consider any feed-forward network with linear connections and ReLU activation functions that outputs logits f k ( x ) and satisﬁes assumptions ( H ).

Original: Suppose that there is a ﬁxed multiset of integers { a 1 , . .
Revised: Suppose that there is a ﬁxed multiset of integers { a 1 , . .

Original: . , a n } such that each path from input to output traverses exactly n average pooling nodes with degrees { a 1 , . .
Revised: . , a n } such that each path from input to output traverses exactly n average pooling nodes with degrees { a 1 , . .

Original: . , a n } .
Revised: . , a n } .

Original: Then:
Revised: Then:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, if the net satisﬁes the symmetry assumption ( S ), then: | ∂ x f k | ∝
Revised: Furthermore, if the net satisﬁes the symmetry assumption ( S ), then: | ∂ x f k | ∝

Original: [Equation]
Revised: [Equation]

Original: Two remarks.
Revised: Two remarks.

Original: First, in all this proof, “weight” encompasses both the standard random weights, and the constant (deterministic) weights equal to 1 / ( in-degree ) of the average-poolings.
Revised: First, in all this proof, “weight” encompasses both the standard random weights, and the constant (deterministic) weights equal to 1 / ( in-degree ) of the average-poolings.

Original: Second, assumption H5 implies that the average-pooling nodes have disjoint input nodes: otherwise, there would be two non-zero deterministic weights w, w (cid:48) from a same neuron that would hence satisfy: E
Revised: Second, assumption H5 implies that the average-pooling nodes have disjoint input nodes: otherwise, there would be two non-zero deterministic weights w, w (cid:48) from a same neuron that would hence satisfy: E

Original: W [ w w (cid:48) ] (cid:54) = 0 .
Revised: W [ w w (cid:48) ] (cid:54) = 0 .

Original: Proof.
Revised: Proof.

Original: As previously, let o designate any ﬁxed output-logit f k ( x ) .
Revised: As previously, let o designate any ﬁxed output-logit f k ( x ) .

Original: For any path p , let a be the set of average-pooling nodes of p and let q be the set of remaining nodes.
Revised: For any path p , let a be the set of average-pooling nodes of p and let q be the set of remaining nodes.

Original: Each path-product ω p satisﬁes: ω p =
Revised: Each path-product ω p satisﬁes: ω p =

Original: ω q ω a , where ω a is a same ﬁxed constant.
Revised: ω q ω a , where ω a is a same ﬁxed constant.

Original: For two distinct paths p , p (cid:48) , Lemma 10 therefore yields: E W (cid:2) ω 2 p (cid:3) =
Revised: For two distinct paths p , p (cid:48) , Lemma 10 therefore yields: E W (cid:2) ω 2 p (cid:3) =

Original: ω
Revised: ω

Original: 2 a E W (cid:2) ω
Revised: 2 a E W (cid:2) ω

Original: 2 q (cid:3) and E W [ ω p ω p (cid:48) ]
Revised: 2 q (cid:3) and E W [ ω p ω p (cid:48) ]

Original: = 0 .
Revised: = 0 .

Original: Combining this with Lemma 9 and under assumption ( S ), we get similarly to (13):
Revised: Combining this with Lemma 9 and under assumption ( S ), we get similarly to (13):

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Therefore, | ∂ x o | = | ∂ x f k | ∝ 1 / (cid:112) d (cid:81) ni
Revised: Therefore, | ∂ x o | = | ∂ x f k | ∝ 1 / (cid:112) d (cid:81) ni

Original: =1 a i .
Revised: =1 a i .

Original: Again, note that, even without assumption ( S ), using (15) and Lemma 8 shows that
Revised: Again, note that, even without assumption ( S ), using (15) and Lemma 8 shows that

Original: [Equation]
Revised: [Equation]

Original: which proves (14).
Revised: which proves (14).

Original: C C OMPARISON TO THE C ROSS -L IPSCHITZ R EGULARIZER
Revised: C C OMPARISON TO THE C ROSS -L IPSCHITZ R EGULARIZER

Original: In their Theorem 2.1, Hein & Andriushchenko (2017) show that the minimal (cid:15) = (cid:107) δ (cid:107) p perturbation to fool the classiﬁer must be bigger than:
Revised: In their Theorem 2.1, Hein & Andriushchenko (2017) show that the minimal (cid:15) = (cid:107) δ (cid:107) p perturbation to fool the classiﬁer must be bigger than:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: They argue that the training procedure typically already tries to maximize f c ( x ) − f k ( x ) , thus one only needs to additionally ensure that (cid:107) ∂ x f c ( x ) − ∂ x f k ( x ) (cid:107) q is small.
Revised: They argue that the training procedure typically already tries to maximize f c ( x ) − f k ( x ) , thus one only needs to additionally ensure that (cid:107) ∂ x f c ( x ) − ∂ x f k ( x ) (cid:107) q is small.

Original: They then introduce what they call a Cross-Lipschitz Regularization, which corresponds to the case p = 2 and involves the gradient differences between all classes:
Revised: They then introduce what they call a Cross-Lipschitz Regularization, which corresponds to the case p = 2 and involves the gradient differences between all classes:

Original: [Equation]
Revised: [Equation]

Original: In contrast, using (10), (the square of) our proposed regularizer (cid:107) ∂ x L(cid:107) q from (4) can be rewritten, for p = q = 2 as:
Revised: In contrast, using (10), (the square of) our proposed regularizer (cid:107) ∂ x L(cid:107) q from (4) can be rewritten, for p = q = 2 as:

Original: [Equation]
Revised: [Equation]

Original: Although both (17) and (18) consist in K 2 terms, corresponding to the K 2 cross-interaction between the K classes, the big difference is that while in (17) all classes play exactly the same role, in (18) the summands all refer to the target class c in at least two different ways.
Revised: Although both (17) and (18) consist in K 2 terms, corresponding to the K 2 cross-interaction between the K classes, the big difference is that while in (17) all classes play exactly the same role, in (18) the summands all refer to the target class c in at least two different ways.

Original: First, all gradient differences are always taken with respect to ∂ x f c .
Revised: First, all gradient differences are always taken with respect to ∂ x f c .

Original: Second, each summand is weighted by the probabilities q k ( x ) and q h ( x ) of the two involved classes, meaning that only the classes with a non-negligible probability get their gradient regularized.
Revised: Second, each summand is weighted by the probabilities q k ( x ) and q h ( x ) of the two involved classes, meaning that only the classes with a non-negligible probability get their gradient regularized.

Original: This reﬂects the idea that only points near the margin need a gradient regularization, which incidentally will make the margin sharper.
Revised: This reﬂects the idea that only points near the margin need a gradient regularization, which incidentally will make the margin sharper.

Original: D P ERCEPTION T HRESHOLD
Revised: D P ERCEPTION T HRESHOLD

Original: To keep the average pixel-wise variation constant across dimensions d , we saw in (3) that the threshold (cid:15) p of an (cid:96) p -attack should scale like d 1 /p .
Revised: To keep the average pixel-wise variation constant across dimensions d , we saw in (3) that the threshold (cid:15) p of an (cid:96) p -attack should scale like d 1 /p .

Original: We will now see another justiﬁcation for this scaling.
Revised: We will now see another justiﬁcation for this scaling.

Original: Contrary to the rest of this work, where we use a ﬁxed (cid:15) p for all images x , here we will let (cid:15) p depend on the (cid:96) 2 -norm of x .
Revised: Contrary to the rest of this work, where we use a ﬁxed (cid:15) p for all images x , here we will let (cid:15) p depend on the (cid:96) 2 -norm of x .

Original: If, as usual, the dataset is normalized such that the pixels have on average variance 1, both approaches are almost equivalent.
Revised: If, as usual, the dataset is normalized such that the pixels have on average variance 1, both approaches are almost equivalent.

Original: Suppose that given an (cid:96) p -attack norm, we want to choose (cid:15) p such that the signal-to-noise ratio (SNR) (cid:107) x (cid:107) 2 / (cid:107) δ (cid:107) 2 of a perturbation δ with (cid:96) p -norm ≤ (cid:15) p is never greater than a given SNR threshold 1 /(cid:15) .
Revised: Suppose that given an (cid:96) p -attack norm, we want to choose (cid:15) p such that the signal-to-noise ratio (SNR) (cid:107) x (cid:107) 2 / (cid:107) δ (cid:107) 2 of a perturbation δ with (cid:96) p -norm ≤ (cid:15) p is never greater than a given SNR threshold 1 /(cid:15) .

Original: For p = 2 this imposes (cid:15) 2 = (cid:15) (cid:107) x (cid:107) 2 .
Revised: For p = 2 this imposes (cid:15) 2 = (cid:15) (cid:107) x (cid:107) 2 .

Original: More generally, studying the inclusion of (cid:96) p -balls in (cid:96) 2 -balls yields
Revised: More generally, studying the inclusion of (cid:96) p -balls in (cid:96) 2 -balls yields

Original: [Equation]
Revised: [Equation]

Original: Note that this gives again (cid:15) p = (cid:15) ∞
Revised: Note that this gives again (cid:15) p = (cid:15) ∞

Original: d 1 /p .
Revised: d 1 /p .

Original: This explains how to adjust the threshold (cid:15) with varying (cid:96) p -attack norm.
Revised: This explains how to adjust the threshold (cid:15) with varying (cid:96) p -attack norm.

Original: Now, let us see how to adjust the threshold of a given (cid:96) p -norm when the dimension d varies.
Revised: Now, let us see how to adjust the threshold of a given (cid:96) p -norm when the dimension d varies.

Original: Suppose that x is a natural image and that decreasing its dimension means either decreasing its resolution or cropping it.
Revised: Suppose that x is a natural image and that decreasing its dimension means either decreasing its resolution or cropping it.

Original: Because the statistics of natural images are approximately resolution and scale invariant (Huang, 2000), in either case the average squared value of the image pixels remains unchanged, which implies that (cid:107) x (cid:107) 2 scales like √ d .
Revised: Because the statistics of natural images are approximately resolution and scale invariant (Huang, 2000), in either case the average squared value of the image pixels remains unchanged, which implies that (cid:107) x (cid:107) 2 scales like √ d .

Original: Pasting this back into (19), we again get:
Revised: Pasting this back into (19), we again get:

Original: [Equation]
Revised: [Equation]

Original: In particular, (cid:15) ∞ ∝ (cid:15) is a dimension-free number, exactly like in (3) of the main part.
Revised: In particular, (cid:15) ∞ ∝ (cid:15) is a dimension-free number, exactly like in (3) of the main part.

Original: Now, why did we choose the SNR as our invariant reference quantity and not anything else?
Revised: Now, why did we choose the SNR as our invariant reference quantity and not anything else?

Original: One reason is that it corresponds to a physical power ratio between the image and the perturbation, which we think the human eye is sensible to.
Revised: One reason is that it corresponds to a physical power ratio between the image and the perturbation, which we think the human eye is sensible to.

Original: Of course, the eye’s sensitivity also depends on the spectral frequency of the signals involved, but we are only interested in orders of magnitude here.
Revised: Of course, the eye’s sensitivity also depends on the spectral frequency of the signals involved, but we are only interested in orders of magnitude here.

Original: Another point: any image x yields an adversarial perturbation δ x , where by constraint (cid:107) x (cid:107) 2 / (cid:107) δ x (cid:107) ≤ 1 /(cid:15) .
Revised: Another point: any image x yields an adversarial perturbation δ x , where by constraint (cid:107) x (cid:107) 2 / (cid:107) δ x (cid:107) ≤ 1 /(cid:15) .

Original: For (cid:96) 2 -attacks, this inequality is actually an equality.
Revised: For (cid:96) 2 -attacks, this inequality is actually an equality.

Original: But what about other (cid:96) p -attacks: (on average over x ,) how far is the signal-to-noise ratio from its imposed upper bound 1 /(cid:15) ?
Revised: But what about other (cid:96) p -attacks: (on average over x ,) how far is the signal-to-noise ratio from its imposed upper bound 1 /(cid:15) ?

Original: For p (cid:54)∈ { 1 , 2 , ∞} , the answer unfortunately depends on the pixel-statistics of the images.
Revised: For p (cid:54)∈ { 1 , 2 , ∞} , the answer unfortunately depends on the pixel-statistics of the images.

Original: But when p is 1 or ∞ , then the situation is locally the same as for p = 2 .
Revised: But when p is 1 or ∞ , then the situation is locally the same as for p = 2 .

Original: Speciﬁcally:
Revised: Speciﬁcally:

Original: Lemma 12.
Revised: Lemma 12.

Original: Let x be a given input and (cid:15) > 0 .
Revised: Let x be a given input and (cid:15) > 0 .

Original: Let (cid:15) p be the greatest threshold such that for any δ with (cid:107) δ (cid:107) p ≤ (cid:15) p , the SNR (cid:107) x (cid:107) 2 / (cid:107) δ (cid:107) 2 is ≤ 1 /(cid:15) .
Revised: Let (cid:15) p be the greatest threshold such that for any δ with (cid:107) δ (cid:107) p ≤ (cid:15) p , the SNR (cid:107) x (cid:107) 2 / (cid:107) δ (cid:107) 2 is ≤ 1 /(cid:15) .

Original: Then (cid:15) p = (cid:15) (cid:107) x (cid:107) 

Revised: Then (cid:15) p = (cid:15) (cid:107) x (cid:107)

Original: d 1 /p − 1 / 2 .
Revised: d 1 /p − 1 / 2 .

Original: Moreover, for p ∈ { 1 , 2 , ∞} , if δ x is the (cid:15) p -sized (cid:96) p -attack that locally maximizes the loss-increase i.e. δ
Revised: Moreover, for p ∈ { 1 , 2 , ∞} , if δ x is the (cid:15) p -sized (cid:96) p -attack that locally maximizes the loss-increase i.e. δ

Original: [Equation]
Revised: [Equation]

Original: | , then:
Revised: | , then:

Original: [Equation]
Revised: [Equation]

Original: and E x [
Revised: and E x [

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The ﬁrst paragraph follows from the fact that the greatest (cid:96) p -ball included in an (cid:96) 2 -ball of radius (cid:15) (cid:107) x (cid:107) 2 has radius (cid:15) (cid:107) x (cid:107) 

Revised: The ﬁrst paragraph follows from the fact that the greatest (cid:96) p -ball included in an (cid:96) 2 -ball of radius (cid:15) (cid:107) x (cid:107) 2 has radius (cid:15) (cid:107) x (cid:107)

Original: d 1 /p − 1 / 2 .
Revised: d 1 /p − 1 / 2 .

Original: The second paragraph is clear for p = 2 .
Revised: The second paragraph is clear for p = 2 .

Original: For p = ∞ , it follows from the fact that δ x = (cid:15) ∞ sign ∂ x L which satisﬁes: (cid:107) δ x (cid:107) 2 = (cid:15) ∞ √ d = (cid:15) (cid:107)
Revised: For p = ∞ , it follows from the fact that δ x = (cid:15) ∞ sign ∂ x L which satisﬁes: (cid:107) δ x (cid:107) 2 = (cid:15) ∞ √ d = (cid:15) (cid:107)

Original: x
Revised: x

Original: (cid:107) 2 .
Revised: (cid:107) 2 .

Original: For p = 1 , it is because δ x = (cid:15) 1 max i =

Revised: For p = 1 , it is because δ x = (cid:15) 1 max i =

Original: d | ( ∂ x L )
Revised: d | ( ∂ x L )

Original: i | , which satisﬁes: (cid:107) δ x (cid:107) 2 = (cid:15) 2 / √ d = (cid:15) (cid:107) x (cid:107) 2 .
Revised: i | , which satisﬁes: (cid:107) δ x (cid:107) 2 = (cid:15) 2 / √ d = (cid:15) (cid:107) x (cid:107) 2 .

Original: Intuitively, this means that for p ∈ { 1 , 2 , ∞} , the SNR of (cid:15) p -sized (cid:96) p -attacks on any input x will be exactly equal to its ﬁxed upper limit 1 /(cid:15) .
Revised: Intuitively, this means that for p ∈ { 1 , 2 , ∞} , the SNR of (cid:15) p -sized (cid:96) p -attacks on any input x will be exactly equal to its ﬁxed upper limit 1 /(cid:15) .

Original: And in particular, the mean SNR over samples x is the same ( 1 /(cid:15) ) in all three cases.
Revised: And in particular, the mean SNR over samples x is the same ( 1 /(cid:15) ) in all three cases.

Revised: E V ULNERABILITY -D IMENSION D EPENDENCE USING D OWNSIZED I MAGE N ET I MAGES

Revised: We also ran a similar experiment as in Section 4.2, but instead of using upsampled CIFAR10 images, we created a 12 -class dataset of approximately 80 , 000 3 × 256 × 256 -sized RGBimages by merging similar ImageNet-classes, resizing the smallest image-edge to 256 pixels and

Revised: [Figure]

Revised: center-cropping the result.

Revised: We then downsized the images to 32 , 64 ,and 256 pixels per edge, and trained, not 1, but 10 CNNs per image-size .

Original: E F IGURES WITH AN (cid:96) 2 P ERTURBATION -T HRESHOLD AND D EEP -F OOL A TTACKS
Revised: We then computed their adversarial vulnerability and average (cid:107) ∂ x L(cid:107) 1 . This gave us 2 values per trained net, i.e. 2 x 10 values per image-size, which are shown in 6. The lines follow their medians, the errorbars show their 10 th and 90 th quantiles. The conclusions are identical to Section 4.2: after usual training, the vulnerability and gradient-norms still increase like √ d .

Revised: Note that, as the gradients get much larger at higher dimensions, the ﬁrst order approximation in (2) becomes less and less valid, which explains the little inﬂection of the adversarial vulnerability curve.

Revised: For smaller (cid:15) -thresholds, we veriﬁed that the inﬂection disappears.

Revised: F F IGURES WITH AN (cid:96) 2 P ERTURBATION -T HRESHOLD AND D EEP -F OOL A TTACKS

Original: Here we plot the same curves as in the main part, but using an (cid:96) 2 -attack threshold of size (cid:15) 2 = 0 .
Revised: Here we plot the same curves as in the main part, but using an (cid:96) 2 -attack threshold of size (cid:15) 2 = 0 .

Original: 005 √ d instead of the (cid:96) ∞ -threshold and deep-fool attacks (Moosavi-Dezfooli et al., 2016) instead of iterative (cid:96) ∞ -ones in Figs.
Revised: 005 √ d instead of the (cid:96) ∞ -threshold and deep-fool attacks (Moosavi-Dezfooli et al., 2016) instead of iterative (cid:96) ∞ -ones in Figs.

Original: 6 and 7.
Revised: 8 and 9.

Original: Note that contrary to (cid:96) ∞ -thresholds, (cid:96) 2 -thresholds must be rescaled by √ d to stay consistent across dimensions (see Eq.3 and Appendix D).
Revised: Note that contrary to (cid:96) ∞ -thresholds, (cid:96) 2 -thresholds must be rescaled by √ d to stay consistent across dimensions (see Eq.3 and Appendix D).

Original: All curves look essentially the same as their counterparts in the main text.
Revised: All curves look essentially the same as their counterparts in the main text.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: r i

Original: [Figure]
Revised: [Figure]

Original: ilit y [Figure] et al., 2016) instead of iterative (cid:96) ∞ ones. All curves look essentially the same than in Fig.
Revised: an (cid:96) ∞ one) and using deep-fool (instead of iterative- (cid:96) ∞ ) attacks to approximate adversarial vulnerability.

Original: F A V ARIANT OF A DVERSARIALLY -A UGMENTED T RAINING
Revised: G A V ARIANT OF A DVERSARIALLY -A UGMENTED T RAINING

Original: In usual adversarially-augmented training, the adversarial image x + δ is generated on the ﬂy, but is nevertheless treated as a ﬁxed input of the neural net, which means that the gradient does not[Figure] k ∂ x L k get backpropagated through δ .
Revised: In usual adversarially-augmented training, the adversarial image x + δ is generated on the ﬂy, but is nevertheless treated as a ﬁxed input of the neural net, which means that the gradient does not get backpropagated through δ .

Original: This need not be.
Revised: This need not be.

Original: As δ is itself a function of x , the gradients could actually also be backpropagated through δ .
Revised: As δ is itself a function of x , the gradients could actually also be backpropagated through δ .

Original: As it was only a one-line change of our code, we used this opportunity to test this variant of adversarial training (FGSM-variant in Figure 2) and thank Martín
Revised: As it was only a one-line change of our code, we used this opportunity to test this variant of adversarial training (FGSM-variant in Figure 2) and thank Martín

Original: Arjovsky for suggesting it.
Revised: Arjovsky for suggesting it.

Original: But except for an increased computation time, we found no signiﬁcant difference compared to usual augmented training.
Revised: But except for an increased computation time, we found no signiﬁcant difference compared to usual augmented training.