I MPROVED S AMPLE C OMPLEXITIES FOR D EEP N ETWORKS AND R OBUST C LASSIFICATION VIA AN A LLL AYER M ARGIN
Anonymous authors
Paper under double-blind review
A BSTRACT
For linear classiﬁers, the relationship between (normalized) output margin and generalization is captured in a clear and simple bound – a large output margin implies good generalization.
Unfortunately, for deep models, this relationship is less clear: existing analyses of the output margin give complicated bounds which sometimes depend exponentially on depth.
In this work, we propose to instead analyze a new notion of margin, which we call the “all-layer margin.”
Our analysis reveals that the all-layer margin has a clear and direct relationship with generalization for deep models.
This enables the following concrete applications of the all-layer margin: 1) by analyzing the all-layer margin, we obtain tighter generalization bounds for neural nets which depend on Jacobian and hidden layer norms and remove the exponential dependency on depth 2) our neural net results easily translate to the adversarially robust setting, giving the ﬁrst direct analysis of robust test error for deep networks, and 3) we present a theoretically inspired training algorithm for increasing the all-layer margin. Our algorithm improves both clean and adversarially robust test performance over strong baselines in practice.
I NTRODUCTION
The most popular classiﬁcation objectives for deep learning, such as cross entropy loss, encourage a larger output margin – the gap between predictions on the true label and and next most conﬁdent label.
These objectives have been popular long before deep learning was prevalent, and there is a long line of work showing they enjoy strong statistical guarantees for linear and kernel methods (Bartlett and Mendelson, 2002;
Koltchinskii et al., 2002; Hofmann et al., 2008; Kakade et al., 2009).
These guarantees have been used to explain the successes of popular algorithms such as SVM (Boser et al., 1992; Cortes and Vapnik, 1995).
For linear classiﬁers, the relationship between output margin and generalization is simple and direct – generalization error is controlled by the output margins normalized by the classiﬁer norm .
Concretely, suppose we have n training data points each with norm 1, and let γ i be the output margin on the i -th example.
With high probability, if the classiﬁer perfectly ﬁts the training data, we obtain
[Equation]
For deeper models, the relationship between output margin and generalization is unfortunately less clear and interpretable.
Known bounds for deep nets normalize the output margin by a quantity that either scales exponentially in depth or depends on complex properties of the network (Neyshabur et al., 2015; Bartlett et al., 2017; Neyshabur et al., 2017b; Golowich et al., 2017; Nagarajan and Kolter, 2019; Wei and Ma, 2019).
This is evidently more complicated than the linear case in (1.1).
These complications arise because for deep nets, it is unclear how to properly normalize the output margin.
In this work, we remedy this issue by proposing a new notion of margin, called "all-layer margin", which we use to obtain simple guarantees like (1.1) for deep models.
Let m i be the alllayer margin for the i -th example.
We can simply normalize it by the sum of the complexities of the weights (often measured by the norms or the covering number) and obtain a bound of the following form:
[Equation]
As the name suggests, the all-layer margin considers all layers of the network simultaneously, unlike the output margin which only considers the last layer.
We note that the deﬁnition of the all-layer margin is the key insight for deriving (1.2) – given the deﬁnition, the rest of the proof follows naturally with standard tools.
(Please see equation (2.2) for a formal deﬁnition of the margin, and
Theorem 2.1 for a formal version of bound (1.2).)
To further highlight the good statistical properties of the all-layer margin, we present three of its concrete applications in this paper.
By relating all-layer margin to output margin and other quantities, we obtain improved generalization bounds for neural nets.
In Section 3, we derive an analytic lower bound on the all-layer margin for neural nets with smooth activations which depends on the output margin normalized by other data-dependent quantities.
By substituting this lower bound into (1.2), we obtain a generalization bound in Theorem 3.1 which avoids the exponential depth dependency and has tighter data-dependent guarantees than (Nagarajan and Kolter, 2019; Wei and Ma, 2019) in several ways.
First, their bounds use the same normalizing quantity for each example, whereas our bounds are tighter and more natural because we use a different normalizer for each training example – the local Lipschitzness for that particular example.
Second, our bound depends on the empirical distribution of some complexity measure computed for each training example.
When these complexities are small for each training example, we can obtain convergence rates faster than 1 / √ n .
We provide a more thorough comparison to prior work in Section 3.
Furthermore, for relu networks, we give a tighter generalization bound which removes the dependency on inverse pre-activations suffered by (Nagarajan and Kolter, 2019), which they showed to be large empirically (see Section B.1).
The techniques of (Wei and Ma, 2019) could not remove this dependency because they relied on smooth activations.
We extend our tools to give generalization bounds for adversarially robust classiﬁcation error which are analogous to our bounds in the standard setting.
In Section 4, we provide a natural extension of our all-layer margin to adversarially robust classiﬁcation.
This allows us to translate our neural net generalization bound, Theorem 3.1, directly to adversarially robust classiﬁcation (see Theorem 4.1).
The resulting bound takes a very similar form as our generalization bound for clean accuracy – it simply replaces the data-dependent quantities in Theorem 3.1 with their worst-case values in the adversarial neighborhood of the training example.
As a result, it also avoids explicit exponential dependencies on depth.
As our bound is the ﬁrst direct analysis of the robust test error, it presents a marked improvement over existing work which analyze loose relaxations of the adversarial error (Khim and Loh, 2018;
Yin et al., 2018).
Finally, our analysis of generalization for the clean setting translates directly to the adversarial setting with almost no additional steps.
This is an additional advantage of our all-layer margin deﬁnition.
We design a training algorithm that encourages a larger all-layer margin and demonstrate that it improves empirical performance over strong baselines.
In Section 5, we apply our regularizer to WideResNet models (Zagoruyko and Komodakis, 2016) trained on the CIFAR datasets and demonstrate improved generalization performance for both clean and adversarially robust classiﬁcation.
We hope that these promising empirical results can inspire researchers to develop new methods for optimizing the all-layer margin and related quantities.
1. A DDITIONAL R ELATED
W ORK
Zhang et al.
; Neyshabur et al.
(2017a) note that deep learning often exhibits statistical properties that are counterintuitive to conventional wisdom.
This has prompted a variety of new perspectives for studying generalization in deep learning, such as implicit or algorithmic regularization (Gunasekar et al., 2017;
Li et al., 2017; Soudry et al., 2018; Gunasekar et al., 2018), new analyses of interpolating classiﬁers (Belkin et al., 2018; Liang and Rakhlin, 2018; Hastie et al., 2019; Bartlett et al., 2019), and the noise and stability of SGD (Hardt et al., 2015; Keskar et al., 2016; Chaudhari et al., 2016).
In this work, we adopt a different perspective for analyzing generalization by studying a novel deﬁnition of margin for deep models which differs from the well-studied notion of output margin.
We hope that our generalization bounds can inspire the design of new regularizers tailored towards deep learning.
Classical results have bounded generalization error in terms of the model’s output margin and the complexity of its prediction (Bartlett and Mendelson, 2002; Koltchinskii et al., 2002), but for deep models this complexity grows exponentially in depth (Neyshabur et al., 2015;
Bartlett et al., 2017; Neyshabur et al., 2017b; Golowich et al., 2017).
Long and Sedghi (2019) prove bounds for convolutional networks which scale with the number of parameters and their distance from initialization.
Recently, Nagarajan and Kolter (2019); Wei and Ma (2019) derived complexity measures in terms of hidden layer and Jacobian norms which avoid the exponential dependence on depth, but their proofs require complicated techniques for controlling the complexity of the output margin.
Neyshabur et al.
(2017a)
; Arora et al.
(2018) also provide complexity measures related to the data-dependent stability of the network, but the resulting bounds only apply to a randomized or compressed version of the original classiﬁer.
We provide a simple framework which derives such bounds for the original classiﬁer.
Novak et al.
; Javadi et al.
(2019) study stability-related complexity measures empirically.
A recent line of work establishes rigorous equivalences between logistic loss and output margin maximization.
Soudry et al.
; Ji and Telgarsky (2018) show that gradient descent implicitly maximizes the margin for linearly separable data, and Lyu and Li (2019) prove gradient descent converges to a stationary point of the max-margin formulation for deep homogeneous networks.
Other works show global minimizers of regularized logistic loss are equivalent to margin maximizers, in linear cases (Rosset et al., 2004) and for deep networks (Wei et al., 2018; Nacson et al., 2019).
A number of empirical works also suggest alternatives to the logistic loss which optimize variants of the output margin (Sun et al., 2014; Wen et al., 2016; Liu; Liang et al., 2017; Cao et al., 2019).
The neural net margin at intermediate and input layers has also been studied.
Elsayed et al.
(2018) design an algorithm to maximize a notion of margin at intermediate layers of the network, and Jiang et al.
(2018) demonstrate that the generalization gap of popular architectures can empirically be predicted using statistics of intermediate margin distributions.
Verma et al.
(2018) propose a regularization technique which they empirically show improves the structure of the decision boundary at intermediate layers.
Sokoli´c et al.
(2017) provide generalization bounds based on the input margin of the neural net, but these bounds depend exponentially on the dimension of the data manifold.
These papers study margins deﬁned for individual network layers, whereas our all-layer margin simultaneously considers all layers.
This distinction is crucial for deriving our statistical guarantees.
A number of recent works provide negative results for adversarially robust generalization (Tsipras et al., 2018; Montasser et al., 2019;
Yin et al., 2018; Raghunathan et al., 2019).
We provide positive results stating that adversarial test accuracy can be good if the adversarial all-layer margin is large on the training data.
Schmidt et al.
(2018) demonstrate that more data may be required for generalization on adversarial inputs than on clean data.
Montasser et al.
(2019) provide impossiblity results for robust PAC learning with proper learning rules, even for ﬁnite VC dimension hypothesis classes.
Zhang et al.
(2019) consider the trade-off between the robust error and clean error.
Farnia et al.
(2018) analyze generalization for speciﬁc adversarial attacks and obtain bounds depending exponentially on depth.
Yin et al.
(2018); Khim and Loh (2018) give adversarially robust generalization bounds by upper bounding the robust loss via a transformed/relaxed loss function, and the bounds depend on the product of weight matrix norms.
Yin et al.
(2018) also show that the product of norms is inevitable if we go through the standard tools of Rademacher complexity and the output margin.
Our adversarial all-layer margin circumvents this lower bound because it considers all layers of the network rather than just the output.
1. N OTATION
We use the notation { a i } ki =1 to refer to a sequence of k elements a i indexed by i .
We will use ◦ to denote function composition: f ◦ g ( x ) =
f ( g ( x )) .
Now for function classes F , G , deﬁne
F ◦ G (cid:44) { f ◦ g : f ∈ F , g ∈ G} .
We use D h to denote the partial derivative operator with respect to variable h , and thus for a function f ( h 1 , h 2 ) , we use D h i f
( h 1 , h 2 ) to denote the partial derivative of f with respect to h i evaluated at ( h 1 , h 2 ) .
We will use (cid:107) · (cid:107) to denote some norm.
For a function f mapping between normed spaces D I , D O with norms (cid:107) · (cid:107) I , (cid:107) · (cid:107) O , respectively, deﬁne (cid:107) f (cid:107) op
(cid:44) sup x ∈D I (cid:107) f
( x ) (cid:107) O (cid:107) x
(cid:107) I , which generalizes the operator norm for linear operators.
Let (cid:107) M (cid:107) fro , (cid:107) M (cid:107) 1 , 1 denote the Frobenius norms and the sum of the absolute values of the entries of M , respectively.
For some set S (often a class of functions), we let N (cid:107)·(cid:107) ( (cid:15), S ) be the covering number of
S in the metric induced by norm (cid:107) · (cid:107) with resolution (cid:15) .
For a function class F , let N ∞ ( (cid:15), F ) denote the covering number of F in the metric d ∞ ( f, (cid:98) f )
=
sup x (cid:107) f
( x ) − (cid:98) f
( x ) (cid:107) .
For a function f and distribution P , we use the notation (cid:107) f (cid:107)
L q ( P ) (cid:44) ( E x ∼ P [ | f ( x ) | q ]) 1 /q .
We bound generalization for a test distribution P given a set of n training samples, P n (cid:44) { ( x i , y i ) } ni =1 where x ∈ D 0 denotes inputs and y ∈ [ l ] is an integer label.
We will also use P n to denote the uniform distribution on these training samples.
For a classiﬁer F : D
→ R l , we use the convention that max y (
cid:48) ∈
[ l ] F ( x ) y (cid:48) is its predicted label on input x .
Deﬁne the 0-1 prediction loss (cid:96) 0-1 ( F ( x ) , y ) to output 1 when F incorrectly classiﬁes x and 0 otherwise.
W ARMUP : S IMPLIFIED A LL -L AYER M ARGIN AND ITS G UARANTEES
Popular loss functions for classiﬁcation, such as logistic and hinge loss, attempt to increase the output margin of a classiﬁer by penalizing predictions that are too close to the decision boundary.
Formally, consider the multi-class classiﬁcation setting with a classiﬁer F : D
→ R l , where l denotes the number of labels.
We deﬁne the output margin on example ( x, y ) by γ ( F ( x ) , y ) (cid:44) max { 0 , F ( x ) y − max y (cid:48) (cid:54) =
y F ( x ) y (cid:48) } .
For shallow models such as linear and kernel methods, the output margin maximization objective enjoys good statistical guarantees (Kakade et al., 2009; Hofmann et al., 2008).
For deep networks, the statistical properties of this objective are less clear: until recently, statistical guarantees depending on the output margin also suffered an exponential dependency on depth (Bartlett et al., 2017;
Neyshabur et al., 2017b).
Recent work removed these dependencies but require technically involved proofs and result in complicated bounds depending on numerous properties of the training data (Nagarajan and Kolter, 2019; Wei and Ma, 2019).
In this section, we introduce a new objective with better statistical guarantees for deep models (Theorem 2.1) and outline the steps for proving these guarantees.
Our objective is based on maximizing a notion of margin which measures the stability of a classiﬁer to simultaneous perturbations at all layers.
Suppose that the classiﬁer F ( x ) =
f k ◦ · · · ◦ f 1 ( x ) is computed by composing k functions f k , . . .
, f 1 , and let δ k , . . .
, δ 1 denote perturbations intended to be applied at each hidden layer.
We recursively deﬁne the perturbed network output F ( x, δ 1 , . . .
, δ k ) by
[Equation]
The all-layer margin will now be deﬁned as the minimum norm of δ required to make the classiﬁer misclassify the input.
Formally, for classiﬁer F , input x , and label y , we deﬁne
[Equation]
subject to arg max y
[Equation]
[Equation]
Note that the constraint that F ( x, δ ) misclassiﬁes x is equivalent to enforcing γ ( F ( x ) , y ) ≤ 0 .
Furthermore, m F is strictly positive if and only if the unperturbed prediction F ( x ) is correct.
Here multiplying δ i by the previous layer norm (cid:107) h
i − 1 ( x, δ ) (cid:107) 2 is important and intuitively balances the relative scale of the perturbations at each layer.
We note that the deﬁnition above is simpliﬁed to convey the main intuition behind our results – to obtain the tightest possible bounds, in Sectionsand 4, we use the slightly more general m F deﬁned in Section A.
Prior works have studied, both empirically and theoretically, the margin of a network with respect to single perturbations at an intermediate or input layer (Sokoli´c et al., 2017; Novak et al., 2018;
Elsayed et al., 2018; Jiang et al., 2018).
Our all-layer margin is better tailored towards handling the compositionality of deep networks because it considers simultaneous perturbations to all layers, which is crucial for achieving its statistical guarantees.
Formally, let F (cid:44) { f k ◦ · · · ◦ f 1 : f i ∈ F i } be the class of compositions of functions from function classes F 1 , . .
, F k .
We bound the population classiﬁcation error for F ∈ F based on the distribution of m F on the training data and the sum of the complexities of each layer, measured via covering numbers.
For simplicity, we assume the covering number of each layer scales as log N (cid:107)·(cid:107) op ( (cid:15), F i ) ≤ (cid:98)C 2 i /(cid:15)
(cid:99) for some complexity C i , which is common for many function classes.
Theorem 2.1 (Simpliﬁed version of Theorem A.1) .
In the above setting, with probability 1 − δ over the draw of the training data, all classiﬁers F ∈ F which achieve training error 0 satisfy
[Equation]
where ζ (cid:44) O (cid:16) log(1 /δ )
+log n n (cid:17) is a low-order term.
In other words, generalization is controlled by the sum of the complexities of the layers and the quadratic mean of 1 /m F on the training set.
Theorem A.1 generalizes this statement to provide bounds which depend on the q -th moment of 1 /m F for any integer q > 0 and converge at rates faster than 1 / √ n .
For neural nets, C i scales with weight matrix norms and 1 /m F can be upper bounded by a polynomial in the Jacobian and hidden layer norms and output margin, allowing us to avoid an exponential dependency on depth when these quantities are well-behaved on the training data.
We will break down the proof of Theorem 2.1 into two simple parts.
The ﬁrst part hinges on showing that m F has low complexity which scales with the sum of the complexities at each layer.
The second part relates m F to the 0-1 loss using the simple fact that m F ( x, y ) is nonzero if and only if F correctly classiﬁes x .
Lemma 2.1 (Complexity Decomposition Lemma) .
Let m ◦ F = { m F : F ∈ F} denote the family of all-layer margins of function compositions in F .
Then
[Equation]
The covering number of an individual layer commonly scales as log N (cid:107)·(cid:107) op ( (cid:15) i , F i ) ≤ (cid:98)C 2 i /(cid:15)
2 i (cid:99) .
In this case, for all (cid:15) > 0 , we obtain log N ∞ ( (cid:15), m ◦ F ) ≤
[Equation]
Lemma 2.1 shows that the complexity of m F scales linearly in depth for any choice of layers F i .
In sharp contrast, lower bounds show that the complexity of the output margin scales exponentially in depth via a product of Lipschitz constants of all the layers (Bartlett et al., 2017; Golowich et al., 2017).
Our proof only relies on basic properties of m F , indicating that m F is naturally betterequipped to handle the compositionality of F .
In particular, we prove Lemma 2.1 by leveraging a uniform Lipschitz property of m F .
This uniform Lipschitz property does not hold for prior deﬁnitions of margin and reﬂects the key insight in our deﬁnition – it arises only because our margin depends on simultaneous perturbations to all layers.
Claim 2.1.
For any two compositions F = f k ◦ · · · ◦ f 1 and (cid:98) F = (cid:98) f k ◦ · · · ◦ (cid:98) f 1 and any ( x, y ) , we have | m F ( x, y ) − m (cid:98) F ( x, y )
| ≤ (cid:113)(cid:80) ki =1 (cid:107) f
i − (cid:98) f i (cid:107) 2 op .
Proof sketch.
Let δ (cid:63) be the optimal choice of δ in the deﬁnition of m F ( x, y ) .
We will construct (cid:98) δ such that (cid:107) (cid:98) δ (cid:107) 2 ≤ (cid:107) δ (cid:63) (cid:107) {+
+}
+ (cid:113)(cid:80) i (cid:107) f
i − (cid:98) f i (cid:107) 2 op and γ ( (cid:98) F ( x, (cid:98) δ ) , y ) = 0 as follows: deﬁne (cid:98) δ i (cid:44) δ (cid:63)i + ∆ i for ∆ i (cid:44) f
i ( h i − 1 ( x,δ (cid:63) )) − (cid:98) f
i ( h i − 1 ( x,δ (cid:63) ))
(cid:107) h
i − 1 ( x,δ (cid:63) ) (cid:107), where h is deﬁned as in (2.1) with respect to the classiﬁer F .
Note that by our deﬁnition of (cid:107) · (cid:107) op , we have (cid:107) ∆ i (cid:107) 2 ≤ (cid:107) f
i − (cid:98) f i (cid:107) op .
Now it is possible to check inductively that (cid:98) F ( x, (cid:98) δ ) =
F ( x, δ (cid:63) ) .
In particular, (cid:98) δ is satisﬁes the misclassiﬁcation constraint in the all-layer margin objective for (cid:98) F .
Thus, it follows that m (cid:98) F ( x, y ) ≤ (cid:107) (cid:98) δ (cid:107) 2 ≤ (cid:107) δ (cid:63) (cid:107) 2 + (cid:107) ∆ (cid:107) 2 ≤ m F ( x, y ) + (cid:113)(cid:80) i (cid:107) f i − (cid:98) f i (cid:107) 2 op , where the last inequality followed from (cid:107) ∆ i (cid:107) 2 ≤
[Equation]
op .
With the same reasoning,we obtain m
[Equation]
x, y [Equation] Given Claim 2.1, Lemma 2.1 follows simply by composing (cid:15) i -covers of F i .
We prove a more general version in Section A (see Lemmas A.1 and A.3.)
The second part of the proof of Theorem 2.1 is to upper bound the 0-1 test error by the test error of some smooth surrogate loss (cid:96) ◦ m F .
A result by Srebro et al.
(2010) shows that generic smooth losses (cid:96) enjoy faster O ( n − 1 ) covergence rates if the empirical loss is low.
We straightforwardly combine Lemma 2.1 with their results to obtain the following generalization bound for (cid:96) ◦ m F :
Lemma 2.2.
Suppose that (cid:96) is a β -smooth loss function taking values in [0 , 1] .
Then in the setting of Theorem 2.1, we have with probability 1 − δ for all F ∈ F :
[Equation]
for some universal constant c > 0 .
To complete the proof of Theorem 2.1, we will choose (cid:96) ◦ m F which upper bounds the 0-1 loss such that the right hand side of (2.4) gives the desired bound.
In Section A, we formalize the proof plan presented here and also deﬁne a slightly more general version of m F used to derive the bounds presented in the following Sections 3 and 4.
Connection to (normalized) output margin Finally, we check that when F is a linear classiﬁer, m F recovers the standard output margin.
Thus, we can view the all-layer margin as an extension of the output margin to deeper classiﬁers.
Example 2.1.
In the binary classiﬁcation setting with a linear classiﬁer F ( x ) = w (cid:62) x where the data x has norm 1 , we have m F ( x, y )
=
max { 0 , yw (cid:62) x } = γ ( F ( x ) , y ) .
For deeper models, the all-layer margin can be roughly bounded by a quantity which normalizes the output margin by Jacobian and hidden layer norms.
We formalize this in Lemma 3.1 and use this to prove our main generalization bound for neural nets, Theorem 3.1.
G ENERALIZATION G UARANTEES FOR N EURAL N ETWORKS
Although the all-layer margin is likely difﬁcult to compute exactly, we can analytically lower bound it for neural nets with smooth activations. In this section, we obtain a generalization bound that depends on computable quantities by substituting this lower bound into Theorem 2.1.
Our bound considers the Jacobian norms, hidden layer norms, and output margin on the training data, and avoids the exponential depth dependency when these quantities are well-behaved, as is the case in practice (Arora et al., 2018; Nagarajan and Kolter, 2019; Wei and Ma, 2019). Prior work (Nagarajan and Kolter, 2019; Wei and Ma, 2019) avoided the exponential depth dependency by considering these same quantities but required complicated proof frameworks.
We obtain a simpler proof with tighter dependencies on these quantities by analyzing the all-layer margin.
The neural net classiﬁer F will be parameterized by r weight matrices { W ( i ) } and compute F ( x ) =
W ( r ) φ ( · · · φ ( W (1) x ) · · · ) for smooth activation φ .
Let d be the largest layer dimension.
We model this neural net by a composition of k = 2 r − 1 layers alternating between matrix multiplications and applications of φ and use the subscript in parenthesis ( i ) to emphasize the different indexing system between weight matrices and all the layers.
We will let s ( i ) ( x ) denote the (cid:107) · (cid:107) 2 norm of the layer preceding the i -th matrix multiplication evaluated on input x , and κ j ← i ( x ) will denote the (cid:107) · (cid:107) op norm of the Jacobian of the j -th layer with respect to the i − 1 -th layer evaluated on x .
The following theorem bounds the generalization error of the network and is derived by lower bounding the all-layer margin in terms the quantities s ( i ) ( x ) , κ j ← i ( x ) , γ ( F ( x ) , y ) .
Theorem 3.1.
Assume that the activation φ has a κ (cid:48) φ -Lipschitz derivative.
Fix reference matrices { A ( i ) , B ( i ) } k i =1 and any integer q > 0 .
With probability 1 − δ over the draw of the training sample
P n , all neural nets F which achieve training error 0 satisfy
[Equation]
where κ NN ( i ) captures a local Lipschitz constant of perturbations at layer i and is deﬁned by
[Equation]
for a secondary term ψ ( i ) ( x, y ) given by
[Equation]
← j ( x )
Here the second and third summations above are over the choices of j, j (cid:48) satisfying the constraints speciﬁed in the summation.
We deﬁne a ( i ) by a ( i ) (cid:44) min {√ d (cid:107) W ( i ) − A
( i ) (cid:107) fro , (cid:107) W ( i ) − B ( i ) (cid:107) 1 , 1 }√ log d + poly ( n − 1 ) and ζ (cid:46) r log n +log(1 /δ )
+ (cid:80) i log( a ( i ) +1) n is a low-order term.
For example, when q = 2 , from (3.1) we obtain the following bound which depends on the second moment of κ NN ( i ) and features the familiar 1 / √ n convergence rate in the training set size.
[Equation]
For larger q , we obtain a faster convergence rate in n , but the dependency on κ NN ( i ) gets larger.
We will outline a proof sketch which obtains a variant of Theorem 3.1 with a slightly worse polynomial dependency on κ NN ( i ) and a ( i ) .
For simplicity we defer the proof of the full Theorem 3.1 to Sections B and C. First, we need to slightly redeﬁne m F so that perturbations are only applied at linear layers (formally, ﬁx δ 2 i = 0 for the even-indexed activation layers, and let δ ( i ) (cid:44)
δ 2 i − 1 index perturbations to the
i -th linear layer).
It is possible to check that Lemma 2.1 still holds since activation layers correspond to a singleton function class { φ } with log covering number 0.
Thus, the conclusion of Theorem 2.1 also applies for this deﬁnition of m F .
Now the following lemma relates this all-layer margin to the output margin and Jacobian and hidden layer norms, showing that m F ( x, y ) can be lower bounded in terms of { κ NN ( i ) ( x, y ) } .
Lemma 3.1.
In the setting above, we have the lower bound m F ( x, y ) ≥
[Equation]
Directly plugging the above lower bound into Theorem 2.1 and choosing C 2 i = 0 , C 2 i − 1 = a ( i ) would give a variant of Theorem 3.1 that obtains a different polynomial in κ NN ( i ) , a ( i ) .
Heuristic derivation of Lemma 3.1 We compute the derivative of F ( x, δ ) with respect to δ ( i ) : D [Equation] [Equation]
[Equation]
[Equation]
x,
[Equation]
[Equation] We abused notation to let D h {+
+} i − 1 ( x,δ ) F ( x, δ ) denote the derivative of F with respect to the 2 i − 1 -th perturbed layer evaluated on input ( x, δ ) .
By deﬁnitions of κ j ← i , s ( i ) and the fact that the output margin is 1-Lipschitz, we obtain (cid:107)
[Equation]
With the ﬁrst order approximation γ ( F ( x, δ ) , y ) ≈ γ ( F ( x ) , y )
+ (cid:80) i D δ
( i ) γ ( F ( x, δ ) , y )
|
[Equation]
{+=
+}
δ
[Equation]
i ) around δ = 0 , we obtain
[Equation]
The right hand side is nonnegative whenever (cid:107) δ
[Equation]
2 ≤
[Equation]
x
[Equation]
ri [Equation]
2 , which would imply that m F ( x, y ) ≥
[Equation]
However, this conclusion is imprecise and non-rigorous because of the ﬁrst order approximation – to make the argument rigorous, we also control the smoothness of the network around x in terms of the interlayer Jacobians, ultimately resulting in the bound of Lemma 3.1.
We remark that the quantities κ NN ( i ) are not the only expressions with which we could lower bound m F ( x, y ) .
Rather, the role of κ NN ( i ) is to emphasize the key term s
( i − 1) ( x ) κ 2 r − 1 ← [Equation] i ( x ) γ ( F ( x ) ,y ) , which measures the ﬁrst order stability of the network to perturbation δ ( i ) and relates the all-layer margin to the output margin.
As highlighted above, if this term is small, m F will be large so long as the network is sufﬁciently smooth around ( x, y ) , as captured by ψ ( i ) ( x, y ) .
Comparison to existing bounds We can informally compare Theorem 3.1 to the existing bounds of (Nagarajan and Kolter, 2019; Wei and Ma, 2019) as follows.
First, the leading term s
[Equation]
of κ NN ( i ) depends on three quantities all evaluated on the same training example , whereas the analogous quantity in the bounds of (Nagarajan and Kolter, 2019; Wei and Ma, 2019) appears as max P n
[Equation]
2 i ( x ) , where each maximum is taken over the entire training set.
As we have (cid:107) κ NN ( i ) (cid:107) L q ( P n ) ≤ max P n κ NN ( i ) ( x, y ) , the term (cid:107) κ NN ( i ) (cid:107) L q ( P n ) in our bound can be much smaller than its counterpart in the bounds of (Nagarajan and Kolter, 2019; Wei and Ma, 2019).
An interpretation of the parameter q is that we obtain fast (close to n − 1 ) convergence rates if the model ﬁts every training example perfectly with large all-layer margin, or we could have slower convergence rates with better dependence on the all-layer margin distribution.
It is unclear whether the techniques in other papers can achieve convergence rates faster than O (1 / √ n ) because their proofs require the simultaneous convergence of multiple data-dependent quantities, whereas we bound everything using the single quantity m F .
Additionally, we compare the dependence on the weight matrix norms relative to n (as the degree of n in our bound can vary).
For simplicitly, assume that the reference matrices A ( i ) are set to 0 .
Our dependence on the weight matrix norms relative to the training set size is, up to logarithmic factors, (cid:16) min [Equation] √
[Equation]
W [Equation] , (cid:107)
[Equation]
(cid:17) [Equation] [Equation] q
[Equation]
2) , which always matches or improves on the dependency obtained by PAC-Bayes methods such as (Neyshabur et al., 2017b; Nagarajan and Kolter, 2019).
Wei and Ma (2019) obtain the dependency (cid:107) W ( i ) (cid:62) (cid:107) 2 , 1 √ n , where (cid:107) W ( i ) (cid:62) (cid:107) 2 , 1 is the sum of the (cid:107) · (cid:107)norms of the rows of W ( i ) .
This dependency on W ( i ) is always smaller than ours.
Finally, we note that Theorem 2.1 already gives tighter (but harder to compute) generalization guarantees for relu networks directly in terms of m F .
Existing work contains a term which depends on inverse pre-activations shown to be large in practice (Nagarajan and Kolter, 2019), whereas m F avoids this dependency and is potentially much smaller.
We explicitly state the bound in Section B.1.
G ENERALIZATION G UARANTEES FOR R OBUST C LASSIFICATION
In this section, we apply our tools to obtain generalization bounds for adversarially robust classiﬁcation.
Prior works rely on relaxations of the adversarial loss to bound adversarially robust generalization for neural nets (Khim and Loh, 2018;
Yin et al., 2018).
These relaxations are not tight and in the case of (Yin et al., 2018), only hold for neural nets with one hidden layer.
To the best of our knowledge, our work is the ﬁrst to directly bound generalization of the robust classiﬁcation error for any network.
Our bounds are formulated in terms of data-dependent properties in the adversarial neighborhood of the training data and avoid explicit exponential dependencies in depth.
Let B adv ( x ) denote the set of possible perturbations to the point x (typically some norm ball around x ).
We would like to bound generalization of the adversarial classiﬁcation loss (cid:96) adv0-1 deﬁned by (cid:96) adv0-1 ( F ( x ) , y ) (cid:44) max x (cid:48) ∈B
adv ( x ) (cid:96) 0-1 ( F ( x (cid:48) ) , y ) .
We prove the following bound which essentially replaces all data-dependent quantities in Theorem 3.1 with their adversarial counterparts.
Theorem 4.1.
Assume that the activation φ has a κ (cid:48) φ -Lipschitz derivative.
Fix reference matrices { A ( i ) , B ( i ) } k i =1 and any integer q > 0 .
With probability 1 − δ over the draw of the training sample
P n , all neural nets F which achieve robust training error 0 satisfy
[Equation]
where κ adv ( i ) is deﬁned by κ adv ( i ) ( x, y ) (cid:44) max x (cid:48)
∈B adv ( x ) κ NN ( i ) ( x (cid:48) , y ) for κ NN ( i ) in (3.2) , and a ( i ) , ζ are deﬁned the same as in Theorem 3.1.
Designing regularizers for robust classiﬁcation based on the bound in Theorem 4.1 is a promising direction for future work.
To prove Theorem 4.1, we simply deﬁne a natural extension to our alllayer margin, and the remaining steps follow in direct analogy to the clean classiﬁcation setting.
We deﬁne the adversarial all-layer margin as the smallest all-layer margin on the perturbed inputs: m adv F ( x, y ) (cid:44) min x (cid:48) ∈B
adv ( x ) m F ( x, y ) .
We note that m adv F ( x, y ) is nonzero if and only if F correctly classiﬁes all adversarial perturbations of x .
Furthermore, the adversarial all-layer margin also satisﬁes the uniform Lipschitz property in Claim 2.1.
Thus, the remainder of the proof of Theorem 4. follows the same steps laid out in Section 2.
As before, we note that Theorem 4.1 requires m F to be the more general all-layer margin deﬁned in Section A.
We provide the full proofs in Section E.
E MPIRICAL A PPLICATION OF THE A LL -L AYER M ARGIN
Inspired by the good statistical properties of the all-layer margin, we design an algorithm which encourages a larger all-layer margin during training.
Letting (cid:96) denote the standard cross entropy loss used in training and Θ the parameters of the network, consider the following objective:
[Equation]
This objective can be interpreted as applying the Lagrange multiplier method to a softmax relaxation of the constraint max y (cid:48) F ( x, δ 1 , . .
, δ k ) y (cid:48)
(cid:54) =
y in the objective for all-layer margin.
If G ( δ, Θ; x, y ) is large, this signiﬁes the existence of some δ with small norm for which F Θ ( x, δ ) suffers large loss, indicating that m F Θ is likely small.
This motivates the following training objective over Θ :
[Equation]
Deﬁne δ (cid:63) Θ ,x,y ∈ arg max δ G ( δ, Θ; x, y ) .
From Danskin’s Theorem, if G ( δ, Θ; x, y ) is continuously differentiable 2 , then we have that the quantity − E P n [ ∇ Θ G ( δ (cid:63) Θ ,x,y , Θ; x, y )] will be a descent direction in Θ for the objective L (Θ) (see Corollary A.2 of (Madry et al., 2017) for the derivation of a similar statement).
Although the exact value δ (cid:63) Θ ,x,y is hard to obtain, we can use a substitute ˜ δ Θ ,x,y found via several gradient ascent steps in δ .
This inspires the following all-layer margin optimization (AMO) algorithm: we ﬁnd perturbations ˜ δ for each example in the batch via gradient ascent steps on G ( δ, Θ; x, y ) .
For each example in the batch, we then compute the perturbed loss (cid:96) ( F Θ ( x, ˜ δ Θ ,x,y )) and update Θ with its negative gradient with respect to these perturbed losses.
This method is formally outlined in the P ERTURBED U PDATE procedure of Algorithm 1.
We use Algorithm 1 to train a WideResNet architecture (Zagoruyko and Komodakis, 2016) on
CIFAR10 and CIFAR100 in a variety of settings. For all of our experiments we use t = 1 , η [Equation] 01 , and we apply perturbations following conv layers in the WideResNet basic blocks.
In Table 1 we report the best validation error achieved during a single run of training, demonstrating that our algorithm indeed leads to improved generalization over the strong WideResNet baseline for a variety of settings.
Additional parameter settings and experiments for the feedforward VGG (Simonyan and Zisserman, 2014) architecture are in Section F.1.
Inspired by our robust generalization bound, we also apply AMO to robust classiﬁcation by extending the robust training algorithm of (Madry et al., 2017).
Madry et al.
(2017) adversarially perturb the training input via several steps of projected gradient descent (PGD) and train using the loss computed on this perturbed input.
At each update, our robust AMO algorithm initializes perturbations δ to 0 for every training example in the batch.
The updates to these δ are performed simultaneously with PGD updates for the adversarial perturbations with the same update rule as Algorithm 1.
procedure P ERTURBED U PDATE (minibatch B = { ( x i , y i ) } bi =1 , current parameters Θ )
Initialize δ i =
for i = 1 , . . .
, b .
for s = 1 , . . .
, t do for all ( x i , y i )
∈
B : do
Update δ i ← (1 − η perturb λ )
δ i + η perturb ∇ δ (cid:96) ( F ORWARD P ERTURB ( x i , δ i , Θ ), y i )
Set update g = ∇ Θ (cid:2) 1 b (cid:80) i (cid:96) ( F ORWARD P ERTURB ( x i , δ i , Θ ), y i ) (cid:3) .
Update Θ ← Θ − η ( g + ∇ Θ R (Θ)) .
(cid:46) R is a regularizer, i.e. weight decay.
function F ORWARD P ERTURB ( x , δ , Θ ) (cid:46)
The net has layers f 1 ( · ; Θ) , . . .
, f r ( · ; Θ) , aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaawith intended perturbations δ (1) , . . .
, δ ( r ) .
Initialize h ← x . for j = 1 , . .
r do
Update h ← f j ( h ; Θ) .
Update h ← h
+ (cid:107) h (cid:107) δ
( j ) .
return h
[Table]
In Table [Table] we report best validation performance for robust AMO and the baseline method of (Madry et al., 2017) on CIFAR10.
Our results demonstrate that our robust AMO algorithm can offer strong improvements in the robust accuracy.
We provide parameter settings in Section F.
[Table]
C ONCLUSION
Many popular objectives in deep learning are based on maximizing a notion of output margin, but unfortunately it is difﬁcult to obtain good statistical guarantees by analyzing this output margin.
In this paper, we design a new all-layer margin which attains strong statistical guarantees for deep models.
Our proofs for these guarantees follow very naturally from our deﬁnition of the margin.
We apply the all-layer margin in several ways: 1) we obtain tighter data-dependent generalization bounds for neural nets 2) for adversarially robust classiﬁcation, we directly bound the robust generalization error in terms of local Lipschitzness around the perturbed training examples [Table] and 3) we design a new algorithm to encourage larger all-layer margins and demonstrate improved performance on real data in both clean and adversarially robust classiﬁcation settings.
We hope that our results prompt further study on maximizing all-layer margin as a new objective for deep learning. [Table]
A G ENERALIZED A LL -L AYER M ARGIN AND M ISSING P ROOFS FOR S ECTION
In this section, we provide proofs for Section 2 in a more general and rigorous setting.
We ﬁrst formally introduce the setting, which considers functions composed of layers which map between arbitrary normed spaces.
Recall that F denotes our classiﬁer from Section 2 computed via the composition f k ◦ · · · ◦ f 1 , For convenience, we overload notation and also let it refer to the sequence of functions { f 1 , . . .
, f k } .
Recall that F denotes the class of all compositions of layers F k , . .
, F 1 , where we let functions in
F i map domains D
i − 1 to D i .
We will ﬁx D k (cid:44) R l , the space of predictions for l classes.
Each space is equipped with norm (cid:107) · (cid:107) (our theory allows the norm to be different for every i , but for simplicity we use the same symbol (cid:107) · (cid:107) for all layers).
As in Section 2, we will use F ( x, δ ) to denote the classiﬁer output perturbed by δ .
It will be useful to deﬁne additional notation for the perturbed function between layers i and j , denoted by f j ← i ( h, δ ) , recursively as follows: f
[Equation]
where we choose f i − 1 ← i
( h, δ ) (cid:44) h .
Note that F ( x, δ ) (cid:44)
f k ← 1 ( x, δ ) , and the notation h
i ( x, δ ) from
Section 2 is equivalent to f i ← 1 ( x, δ ) .
We will use the simpliﬁed notation f
j ← i ( x ) (cid:44) f
j ← i ( x, 0) when the perturbation δ is 0 at all layers.
For a given F , we now deﬁne the general all-layer margin
m F : D 0 ×
[ l ] → R as follows:
[Equation]
subject to γ ( F [Equation] , δ [Equation]
A
[Equation]
[Equation]
The norm ||| · ||| will have the following form:
[Equation]
where α i ≥ 0 will be parameters chosen later to optimize the resulting bound, and (cid:107) · (cid:107) p denotes the standard (cid:96) p -norm.
For F = { f 1 , . . .
, f k } , we overload notation and write ||| F ||| =
(cid:107) ( α 1 (cid:107) f 1 (cid:107) op , . . .
, α k (cid:107) f k (cid:107) op ) (cid:107) p .
This more general deﬁnition of m F will be useful for obtaining Theorems 3.1 and 4.1.
Note that by setting α i = 1 for all i and p = 2 , we recover the simpler m F deﬁned in Section 2.
As before, it will be convenient for the analysis to assume that the (cid:15) -covering number of F i in operator norm scales with (cid:15) − 2 .
We formally state this condition for general function classes and norms below:
Condition A.1 ( (cid:15) − 2 covering condition) .
We say that a function class G satisﬁes the (cid:15) − 2 covering condition with respect to norm (cid:107) · (cid:107) with complexity C
(cid:107)·(cid:107) ( G ) if for all (cid:15) > 0 , log N (cid:107)·(cid:107) ( (cid:15), [Equation]
Now we provide the analogue of Theorem 2.1 for the generalized all-layer margin:
Theorem A.1.
Fix any integer q > 0 .
Suppose that all layer functions F i satisfy Condition A.1 with operator norm (cid:107) · (cid:107) op and complexity function C (cid:107)·(cid:107)
op ( F i ) .
Let the all layer margin m F be deﬁned as in (A.1) .
Then with probability 1 − δ over the draw of the training data, all classiﬁers F ∈ F which achieve training error 0 satisfy
[Equation]
where C |||·||| ( F ) (cid:44) (cid:16)(cid:80) i
α 2 p/ ( p +2)
i
C [Equation] op ( F i ) [-
-]2 p/ [Equation]
p [Equation] 2) (cid:17) ( p +2) / 2 p is a complexity (in the sense of Condition A.1) for covering F in ||| · ||| and ζ (cid:44) O (cid:16) log(1 /δ )
+log n n (cid:17) is a low-order term.
Note that this recovers Theorem 2.1 when α i = 1 for all i and p = 2 .
The proof of Theorem A. mirrors the plan laid out in Section 2.
As before, the ﬁrst step of the proof is showing that m F has low complexity as measured by covering numbers.
Lemma A.1.
Deﬁne m ◦ F (cid:44) { ( x, y )
(cid:55)→ m F ( x, y ) : F ∈ F} .
Then
[Equation]
As in Section 2, we prove Lemma A.1 by bounding the error between m F and m (cid:98) F in terms of the ||| · ||| -norm of the difference between F and (cid:98) F .
Claim A.1.
For any x, y ∈ D 0 ×
[ l ] , and function sequences F = { f i } ki =1 , (cid:98) F = { (cid:98) f i } ki =1 , we have | m F ( x, y ) − m (cid:98) F ( x, y )
| ≤ ||| F − (cid:98) F ||| .
Proof.
Suppose that δ (cid:63) optimizes equation (A.1) used to deﬁne m F ( x, y ) .
Now we use the notation h (cid:63)i (cid:44)
f i ← 1 ( x, δ (cid:63) ) .
Deﬁne (cid:98) δ as follows:
[Equation]
We ﬁrst argue via induction that (cid:98) f i ← 1 ( x, (cid:98) δ )
=
h (cid:63)i .
As the base case, we trivially have (cid:98) f 0 ← 1 ( x, (cid:98) δ ) =
x = h
[Equation]
[Equation]
f
[Equation]
the inductive hypothesis)
[Equation]
deﬁnition
[Equation]
deﬁnition of
[Equation]
Thus, we must have (cid:98) F ( x, (cid:98) δ ) =
F ( x, δ (cid:63) ) , so it follows that γ ( (cid:98) F ( x, (cid:98) δ ) , y ) ≤ 0 as well.
Furthermore, by triangle inequality
[Equation]
Now we note that as
[Equation]
f i ( h
[Equation]
− 1 ) (cid:107) (cid:107) h
(cid:63)i − 1 (cid:107) ≤ (cid:107) f
i − (cid:98) f i (cid:107) op , it follows that
[Equation]
||| Thus, using (A.2) and the deﬁnition of m (cid:98) F , we have
[Equation]
where we relied on the fact that ||| δ (cid:63) ||| =
m F ( x, y ) .
Using the same reasoning, we also obtain the inequality m F ( x, y ) ≤ m (cid:98) F ( x, y )
+ ||| F − (cid:98) F ||| .
Combining gives | m F ( x, y ) − m (cid:98) F ( x, y ) | ≤ ||| F − (cid:98) F ||| .
Lemma A.1 now directly follows.
Proof of Lemma A.1.
As Claim A.1 holds for any choice of ( x, y ) ∈ D 0 ×
[ l ] , it follows that if (cid:98) F covers F in norm ||| · ||| , then m ◦ (cid:98) F will be a cover for m ◦ F in the functional ∞ norm.
We now state the generalized version of Lemma 2.2.
The statement below is a straightforward application of our covering number bound in Lemma A.1 with theory in (Srebro et al., 2010); for minor technical reasons we translate their result to covering numbers and reprove it in Section A.1.
Lemma A.2 (Straightforward adaptation from (Srebro et al., 2010)) .
Suppose that (cid:96) is a β -smooth loss function taking values in [0 , 1] .
Furthermore suppose that F satisﬁes Condition A.1 with respect to norm ||| · ||| and complexity C |||·||| ( F ) .
Then with probability 1 − δ , for all F ∈ F ,
[Equation] [Equation]
for some universal constant c > 0 .
The ﬁnal ingredient is showing that when each individual layer F i satisﬁes Condition A.1 in operator norm, the class of compositions F satisﬁes Condition A.1 with respect to norm ||| · ||| .
Lemma A.3.
Suppose that each F i satisﬁes Condition A.1 with norm (cid:107) · (cid:107) op and complexity
C
[Equation]
op ( F i ) .
Deﬁne the complexity measure C |||·||| ( F ) by
[Equation]
Then we have
[Equation]
which by deﬁnition implies that F satisﬁes Condition A.1 with norm ||| · ||| and complexity C |||·||| ( F ) .
Proof.
Let (cid:98) F i be an (cid:15) i -cover of F i in the operator norm (cid:107)·(cid:107) op .
We will ﬁrst show that (cid:98) F (cid:44) { (cid:98) F k ◦· · ·◦ (cid:98) F 1 : (cid:98) F i ∈ (cid:98) F i } is a (cid:107){
α i (cid:15) i } ki =1 (cid:107) p -cover of F in ||| · ||| .
To see this, for any F = ( f k , . . .
, f 1 ) ∈ F , let (cid:98) f i ∈ (cid:98) F i be the cover element for f i , and deﬁne (cid:98) F (cid:44) ( (cid:98) f k , . . .
, (cid:98) f 1 ) .
Then we have
[Equation]
as desired.
Furthermore, we note that log | (cid:98) F| ≤ (cid:80)
i
[Equation]
by Condition A.1.
Now we will choose
[Equation]
We ﬁrst verify that this gives an (cid:15) -cover of F in ||| · ||| :
[Equation]
Next
[Equation]
we check that the covering number is bounded by
[Equation]
i
[Equation]
Finally, we prove Theorem A.1 (and as a result, Theorem 2.1).
This will hinge on applying
Lemma A.2 with the correct choice of smooth loss.
Proof of Theorems A.1 and 2.1.
Deﬁne (cid:96) β ( m ) (cid:44) 1 + 2 min { 0 , Pr Z ∼N (0 , 1) ( Z/ √ β ≥ m ) − 0 . 5 } .
By Claim A.2, this loss is c 1 β smooth and for m F ( x, y ) > 0 satisﬁes
[Equation]
for universal constants c 1 , c 2 .
We additionally have (cid:96) 0-1 ( F ( x ) , y ) ≤ (cid:96) β ( m F ( x, y )) .
Because of
Lemma A.3, the conditions of Lemma A.2 are satisﬁed, and applying Lemma A.2 with smooth loss (cid:96) β gives with probability 1 − δ , for all F ∈ F with training error
[Equation]
where ˜ E F ( β ) is deﬁned by
[Equation]
n and C 2 |||·||| ( F ) is deﬁned as in Lemma A.3.
Choosing β to minimize the above expression would give the desired bound – however, such a post-hoc analysis cannot be performed because the optimal β depends on the training data, and the loss class has to be ﬁxed before the training data is drawn.
Instead, we utilize the standard technique of union bounding over a grid of (cid:98) β in log-scale.
Let ξ (cid:44) C 2 |||·||| ( F ) poly ( n − 1 ) denote the minimum choice of (cid:98) β in this grid, and select in this grid all choices of (cid:98) β in the form ξ 2 j for j ≥ 0 .
For a given choice of (cid:98) β , we assign it failure probability (cid:98) δ =
δ 2 (cid:98) β/ξ , such that by design (cid:80) (cid:98)
δ = δ .
Thus, applying Lemma A.2 for each choice of (cid:98) β with corresponding failure probability (cid:98) δ , we note with probability 1 − δ ,
[Equation]
holds for all (cid:98) β and F ∈ F .
Now for ﬁxed F ∈ F , let β (cid:63)F denote the optimizer of ˜ E F ( β ) .
We claim either there is some choice of (cid:98) β with
[Equation]
or ˜ E F ( β (cid:63) F ) (cid:38) 1 , in which case the generalization guarantees of Theorem A.1 for this F anyways trivially hold.
To see this, we note that there is (cid:98) β in the grid such that (cid:98) β ∈ [ β (cid:63)F , 2 β (cid:63)F + ξ ] .
Then
[Equation]
Furthermore, we note that if β (cid:63)F > poly ( n ) ξ , then ˜ E F ( β (cid:63)F ) (cid:38) 1 .
This allows to only consider (cid:98) β < poly ( n ) ξ , giving (A.4).
Thus, with probability 1 − δ , for all F ∈ F , we have
[Equation]
It remains to observe that setting β (cid:63)F =
Θ
[Equation]

 gives us the theorem statement.
Claim A.2.
For β > 0 , deﬁne the loss function (cid:96) β ( m ) (cid:44) 1 + 2 min { 0 , Pr Z ∼N (0 , 1) ( Z/ √ β ≥ m ) − 0 . 5 } .
Then (cid:96) β satisﬁes the following properties:
For all m ∈ R , (cid:96) β ( m ) ∈ [0 , 1] , and for m ≤ 0 , (cid:96) β ( m ) = 1 . 2.
The function (cid:96) β is c 1 β -smooth for some constant c 1 independent of β . 3.
For any integer q > 0 and m > 0 , (cid:96) β ( m ) ≤ q q/ 2 c q 2 β q/ 2 m q for some constant c 2 independent of q .
Proof.
The ﬁrst property follows directly from the construction of (cid:96) β .
For the second property, we ﬁrst note that
[Equation]
Now ﬁrst note that at m = 0 , the above quantity evaluates to 0 , and thus (cid:96) β has a second derivative everywhere (as m = 0 is the only point where the function switches).
Furthermore,
[Equation]
some constant [Equation] independent of β
[Equation] Thus, the above expression is upper bounded by β √ 2 π c (cid:48) , giving the second property.
For the third property, we note that for m > 0 , (cid:96) β ( m )
=
2 Pr Z ∼N (0 , 1) ( Z/ √ β ≥ m ) .
As the q -th moment of a Gaussian random variable with variance 1 is upper bounded by q q/ 2 c q 2 for all q and some c 2 independent of q , Markov’s inequality gives the desired result.
A. P ROOF OF L EMMA A
The proof is a straightforward application of Lemma A.1 and conversion of (Srebro et al., 2010) from the language of Rademacher complexity to covering numbers.
Proof of Lemma A.2.
We can follow the proof of Theorem 1 in (Srebro et al., 2010), with the only difference that we replace their Rademacher complexity term with our complexity function C |||·||| ( F ) .
For completeness, we outline the steps here.
Deﬁne H ( µ ) = { h ∈ [Equation]
m ◦F : E P n [ h ] ≤ µ } to be the class of functions in (cid:96) ◦ m ◦F with empirical loss at most µ .
Deﬁne ψ ( µ ) [Equation] n log n .
By Claim A.3, the following holds for all µ :
[Equation]
Now using the same steps as (Srebro et al., 2010) (which relies on applying Theorem 6.1 of (Bousquet, 2002)), we obtain for all F ∈ F , with probability 1 − δ
E P [ (cid:96) ◦ m F ] ≤ E P n [ (cid:96) ◦ m F ] + 106 r (cid:63)n +
[Equation]
where r (cid:63)n is the largest solution of ψ ( µ ) = µ .
We now plug in r (cid:63)n (cid:46) β log 2 n C 2 |||·||| ( F ) n and use the fact that √ c 1 c 2 ≤ ( c 1 + c 2 ) / 2 for any c 1 , c 2 > 0 to simplify the square root term in A.5.
[Equation]
for some universal constant c .
Claim A.3.
In the setting above, for all µ > 0 , we have
[Equation]
where { σ i } n i =1 are i.i.d.
Rademacher random variables.
Proof.
First, by Dudley’s Theorem (Dudley, 1967), we have
[Equation]
Now by Claim A.4, we obtain
E
[Equation]
by Claim A
[Equation] [Equation]
We obtained the last line via change of variables to (cid:15) (cid:48)
= (cid:15)/ √ 48 βµ .
Now we substitute α =
[Equation]
|
[Equation]
|
[Equation]
F ) √ 48 βµ √ n
and note that the integrand is 0 for (cid:15) (cid:48)
> C |||·||| ( F ) to get
[Equation]
The following claim applies Lemma A.1 in order to bound the covering number of H ( µ ) in terms of
C |||·||| ( F ) .
Claim A.4.
In the setting of Lemma A.2, we have the covering number bound
[Equation]
Proof.
As (cid:96) ◦ m ◦ F is the composition of a β -smooth loss (cid:96) with the function class m ◦ F , by equation (22) of (Srebro et al., 2010)
we have
[Equation]
A
[Equation]
[Equation]
Condition A
[Equation]
[Equation]
B P ROOFS FOR N EURAL N ET G ENERALIZATION
This section will derive the generalization bounds for neural nets in Theorem 3.1 by invoking the more general results in Section C. Theorem 3.1 applies to all neural nets, but to obtain it, we ﬁrst need to bound generalization for neural nets with ﬁxed norm bounds on their weights (this is a standard step in deriving generalization bounds).
The lemma below states the analogue of Theorem 3.1, for all neural nets satisfying ﬁxed norm bounds on their weights.
Lemma B.1.
In the neural network setting, suppose that the activation φ has a κ (cid:48) φ -Lipschitz derivative.
For parameters { a ( i ) } ri =1 meant to be norm constraints for the weights, deﬁne the class of neural nets with bounded weight norms with respect to reference matrices { A ( i ) , B ( i ) } as follows:
[Equation]
Then with probability 1 − δ , for any q > 0 and for all F ∈ F , we have
[Equation]
where S n denotes the subset of examples classiﬁed correctly by F and κ NN ( i ) is deﬁned as in (3.2) .
Proof.
We will identify the class of neural nets with matrix norm bounds { a ( i ) } ri =1 with a sequence of function families
[Equation]
and let F (cid:44) F 2 r − 1 ◦ · · · ◦ F 1 denote all possible parameterizations of neural nets with norm bounds { a ( i ) } ri =1 .
Let (cid:107)·(cid:107) op be deﬁned with respect to Euclidean norm (cid:107)·(cid:107) 2 on the input and output spaces, which coincides with matrix operator norm for linear operators.
We ﬁrst claim that
[Equation]
op ( (cid:15), F 2 i − 1 ) (cid:46) (cid:22) a ( i ) (cid:15)(cid:23)
This is because we can construct two covers: one for { h (cid:55)→
Wh : √ d (cid:107) W (cid:107) F / √ log d ≤ a ( i ) } , and one for { h (cid:55)→
Wh : (cid:107) W (cid:107) 1 , 1 / √ log d ≤ a ( i ) } , each of which has log size bounded by O ( (cid:98) a ( i )2 /(cid:15) 2 (cid:99) ) by Lemma B.2 and Claim B.2.
Now we offset the ﬁrst cover by the linear operator A ( i ) and the second by B ( i ) and take the union of the two, obtaining an (cid:15) -cover for F 2 i − 1 in operator norm.
Furthermore, log N (cid:107)·(cid:107) op ( (cid:15), F 2 i )
=
simply because F 2 i is the singleton function.
Thus, F 2 i − 1 , F 2 i satisfy Condition A.1 with norm (cid:107) · (cid:107) op and complexity functions
C (cid:107)·(cid:107) op ( F 2 i − 1 ) (cid:46) a ( i ) and C (cid:107)·(cid:107) op ( F 2 i )
= 0 , so we can apply Theorem C.1.
It remains to argue that κ (cid:63) 2 i − 1 ( x, y ) as deﬁned for Theorem C.1 using standard Euclidean norm (cid:107) · (cid:107) 2 is equivalent to κ NN ( i ) ( x, y ) deﬁned in (3.2).
To see this, we note that functions in F 2 j − 1 have 0-Lipschitz derivative, leading those terms with a coefﬁcient of κ (cid:48)
2 j − 1 to cancel in the deﬁnition of κ (cid:63)i ( x, y ) .
There is a 1-1 correspondence between the remaining terms of κ (cid:63) 2 i − 1 ( x, y ) and κ NN ( i ) ( x, y ) , so we can substitute κ NN ( i ) ( x, y ) into Theorem C.1 in place of κ (cid:63) 2 i − 1 ( x, y ) .
Furthermore, as we have C (cid:107)·(cid:107) op ( F 2 i ) = 0 , the corresponding terms disappear in the bound of Theorem C.1, ﬁnally giving the desired result.
Now we obtain Theorem 3.1 by union bounding Lemma B.1 over choices of { a ( i ) } ri =1 .
Proof of Theorem 3.1.
We will use the standard technique of applying Lemma B.1 over many choices of { a ( i ) } , and union bounding over the failure probability.
Choose ξ = poly ( n − 1 ) and consider a grid of { (cid:98) α ( i ) } with (cid:98) a ( i ) =
ξ 2 j i for j i ≥ 1 . We apply Lemma B.1 with for all possible norm bounds { (cid:98) α
( i ) } in the grid, using failure probability (cid:98) δ = δ/ ( (cid:81) i (cid:98) a ( i ) /ξ )
for a given choice of { (cid:98) α ( i ) } .
By union bound, with probability 1 − (cid:80) (cid:98) δ = 1 − δ , the bound of Lemma B.1 holds simultaneously for all choices of { (cid:98) α ( i ) } .
In particular, for the neural net F with parameters { W ( i ) } , there is a choice of { (cid:98) α ( i ) } satisfying
[Equation]
for all i .
The application of Lemma B.1 for this choice of (cid:98) α ( i ) gives us the desired generalization bound.
B. G ENERALIZATION B OUND FOR R ELU N ETWORKS
In the case where φ is the relu activation, we can no longer lower bound the all-layer margin m F ( x, y ) using the techniques in Section C, which rely on smoothness.
However, we can still obtain a generalization bound in terms of the distribution of 1 /m F ( x, y ) on the training data.
We can expect 1 /m F ( x, y ) to be small in practice because relu networks typically exhibit stability to perturbations.
Prior bounds for relu nets suffer from some source of looseness: the bounds of (Bartlett et al., 2017; Neyshabur et al., 2017b) depended on the product of weight norms divided by margin, and the bounds of (Nagarajan and Kolter, 2019) depended on the inverse of the pre-activations, observed to be large in practice.
Our bound avoids these dependencies, and in fact, it is possible to upper bound our dependency on 1 /m F ( x, y ) in terms of both these quantities.
For this setting, we choose a ﬁxed ||| · ||| deﬁned as follows: if i corresponds to a linear layer in the network, set α i = 1 , and for i corresponding to activation layers, set α i = ∞ (in other words, we only allow perturbations after linear layers).
We remark that we could use alternative deﬁnitions of ||| · ||| , but because we do not have a closed-form lower bound on m F , the tradeoff between these formulations is unclear.
Theorem B.1.
In the neural network setting, suppose that φ is any activation (such as the relu function) and m F is deﬁned using ||| · ||| as described above.
Fix any integer q > 0 .
Then with probability 1 − δ , for all relu networks F parameterized by weight matrices { W ( i ) } ri =1 that achieve training error 0, we have
[Equation]
where a ( i ) is deﬁned as in Theorem 3.1, and ζ (cid:44) O (cid:16) log(1 /δ )
+ r log n + (cid:80) i ( a ( i ) +1) n (cid:17) is a low-order term.
The proof follows via direct application of Theorem A.1 and the same arguments as Lemma B. relating matrix norms to covering numbers.
We remark that in the case of relu networks, we can upper bound 1 m F ( x,y ) via a quantity depending on the inverse pre-activations that mirrors the bound of Nagarajan and Kolter (2019).
However, as mentioned earlier, this is a pessimistic upper bound as Nagarajan and Kolter (2019) show that the inverse preactivations can be quite large in practice.
B. M ATRIX C OVERING L EMMAS
In this section we present our spectral norm cover for the weight matrices, which is used in Section B to prove our neural net generalization bounds. Lemma B.2. Let M fro ( B ) denote the set of d 1 ×
d 2 matrices with Frobenius norm bounded by B , i.e.
[Equation]
Then letting d (cid:44) max { d 1 , d 2 } denote the larger dimension, for all (cid:15) > 0 , we have
[Equation]
Proof.
The idea for this proof is that since the cover is in spectral norm, we only need to cover the top d (cid:48) (cid:44) (cid:98) B 2 /(cid:15) 2 (cid:99) singular vectors of matrices M ∈ M .
First, it sufﬁces to work with square matrices, as a spectral norm cover of max { d 1 , d 2 } × max { d 1 , d 2 } matrices will also yield a cover of d 1 × d 2 matrices in spectral norm (as we can extend a d 1 × d 2 matrices to a larger square matrix by adding rows or columns with all 0).
Thus, letting d (cid:44) max { d 1 , d 2 } , we will cover M fro ( B ) deﬁned with respect to d × d matrices.
Let d (cid:48) (cid:44) (cid:98) 9 B 2 /(cid:15) 2 (cid:99) .
We ﬁrst work in the case when d (cid:48) ≤ d .
Let (cid:98) U be a (cid:15) U Frobenius norm cover of d × d (cid:48) matrices with Frobenius norm bound d (cid:48) .
Let (cid:98) V be the cover of d (cid:48)
× d matrices with Frobenius norm bound B in Frobenius norm with resolution (cid:15) V .
We construct a cover (cid:99) M for M fro ( B ) as follows: take all possible combinations of matrices (cid:98) U, (cid:98) V from (cid:98) U , (cid:98) V , and add (cid:98) U (cid:98) V to (cid:99) M .
First note that by Claim B.1, we have
[Equation]
Now we analyze the cover resolution of (cid:99) M : for M ∈ M , ﬁrst let trunc d (cid:48)
( M ) be the truncation of
M to its d (cid:48) largest singular values.
Note that as M has at most d (cid:48) singular values with absolute value greater than (cid:15)/ 3 , (cid:107) M − trunc d (cid:48)
( M ) (cid:107) op ≤ (cid:15)/ 3 .
Furthermore, let USV = trunc d (cid:48)
( M ) be the SVD decomposition of this truncation, where U ∈ R d × d (cid:48) , (cid:107) U (cid:107) fro ≤ d (cid:48) and SV ∈ R d (cid:48)
× d , (cid:107) SV (cid:107) fro ≤ B .
Let (cid:98) U ∈ (cid:98)
U satisfy (cid:107) (cid:98)
U − U (cid:107) fro ≤ (cid:15) U , and (cid:98) V ∈ (cid:98) V satisfy (cid:107) (
cid:98) V − SV (cid:107) fro ≤ (cid:15) V .
Let (cid:99)
M = (cid:98) U (cid:98) V .
Then we obtain
[Equation]
Thus, setting (cid:15) U =
(cid:15)/ 3 B , (cid:15) V =
(cid:15)/ 3 d (cid:48) , then we get a (cid:15) -cover of M with log cover size (cid:98) 9 dB 2 /(cid:15) 2 (cid:99) (log 81 d (cid:48) 2 B 2 /(cid:15) 2 ) .
As d (cid:48) ≤ d , this simpliﬁes to (cid:98) 36 dB 2 log(9 d ) /(cid:15) 2 (cid:99) .
Now when d (cid:48) ≥ d , we simply take a Frobenius norm cover of d × d matrices with Frobenius norm bound B , which by Claim B.1 has log size at most d 2 log(3 B/(cid:15) ) ≤ (cid:98) 36 dB 2 log(9 d ) /(cid:15) 2 (cid:99) , where the inequality followed because 9 B 2 /(cid:15) 2 ≥ d .
Combining both cases, we get for all (cid:15) > 0 ,
[Equation]
The following claims are straightforward and follow from standard covering number bounds for (cid:107) · (cid:107) 2 and (cid:107) · (cid:107) 1 balls.
Claim B.1.
Let M fro ( B ) denote the class of d 1 × d 2 matrices with Frobenius norm bounded by B .
Then for 0 < (cid:15) < B , log N (cid:107)·(cid:107) fro ( (cid:15), M fro ( B ))
≤ d 1 d 2 log(3 B/(cid:15) ) .
Claim B.2.
Let M (cid:107)·(cid:107) 1 , 1 ( B ) denote the class of d 1 × d 2 matrices with the (cid:96) 1 norm of its entries bounded by B .
Then log N (cid:107)·(cid:107) fro ( (cid:15), M (cid:107)·(cid:107) 1 , 1 ( B )) ≤ 5 (cid:98) B 2 /(cid:15) 2 (cid:99) log 10 d .
C G ENERALIZATION B OUND FOR S MOOTH F UNCTION C OMPOSITIONS
In this section, we present the bound for general smooth function compositions used to prove Theorem 3.1.
We will work in the same general setting as Section A. Let J j ← i ( x, δ ) denote the i -to- j Jacobian evaluated at f i − 1 ← 1 ( x, δ ) , i.e. J j ← i ( x, δ ) (cid:44) D
h f
j ← i ( h, δ ) | h
= f i − 1 ← 1 ( x,δ ) .
We will additionally deﬁne general notation for hidden layer and Jacobian norms which coincides with our notation for neural nets.
Let s i ( x ) (cid:44) (cid:107) f i ← 1 ( x ) (cid:107) and s 0 ( x ) (cid:44) (cid:107) x
(cid:107) .
As the function Df j ← i outputs operators mapping D i − 1 to D j , we can additionally deﬁne κ j ← i ( x ) (cid:44) (cid:107) Df j ← i ◦ f i − 1 ← 1 ( x ) (cid:107) op , with κ j ← j +1 ( x ) (cid:44) 1 .
Let κ (cid:48)
i be an upper bound on the Lipschitz constant of Df i ← i measured in operator norm:
[Equation]
Now we deﬁne the value κ (cid:63)i ( x, y ) , which can be thought of as a Lipschitz constant for perturbation δ i in the deﬁnition of m F , as follows:
[Equation]
For this general setting, the following theorem implies that for any integer q > 0 , if F classiﬁes all training examples correctly, then its error converges at a rate that scales with n − q/ ( q +2) and the products (cid:107) κ (cid:63)i (cid:107) L q ( P n ) C (cid:107)·(cid:107) op ( F i ) .
Theorem C.1.
Let F = { f k ◦ · · · ◦ f 1 : f i ∈ F i } denote a class of compositions of functions from k families {F i } k i =1 , each of which satisﬁes Condition A.1 with operator norm (cid:107) · (cid:107) op and complexity
C
[Equation]
op ( F i ) .
For any choice of integer q > 0 , with probability 1 − δ for all F ∈ F the following bound holds:
[Equation]
where S n denotes the subset of training examples correctly classiﬁed by F and κ (cid:63)i is deﬁned in (C.1) .
In particular, if F classiﬁes all training samples correctly, i.e. |S n | = n , with probability 1 − δ we have
[Equation]
where ζ (cid:44) O (cid:16) k log n +log(1 /δ )
n (cid:17) is a low order term.
To prove this theorem, we will plug the following lower bound on m F into Lemma A.2 with the appropriate choice of smooth loss (cid:96) , and pick the optimal choice of { α i } ki =1 for the resulting bound.
We remark that we could also use Theorem A.1 as our starting point, but this would still require optimizing over { α i } k i =1 .
Lemma C.1 (General version of Lemma 3.1) .
In the setting of Theorem C.1, where each layer F i is a class of smooth functions, if γ ( F ( x ) , y ) > 0 , we have
[Equation]
We prove Lemma C.1 in Section D by formalizing the intuition outlined in Section 3.
With
Lemma C.1 in hand, we can prove Theorem C.1.
This proof will follow the same outline as the proof of Theorem A.1.
The primary difference is that we optimize over k values of α i , whereas
Theorem A.1 only optimized over the smoothness β .
Proof of Theorem C.1.
We use (cid:96) β with β = 1 deﬁned in Claim A.2 as a surrogate loss for the 0loss.
Since Claim A.2 gives (cid:96) 0-1 ( F ( x ) , y ) ≤ (cid:96) β
=1 ( m F ( x, y )) , by Lemma A.2 it follows that
E P [ (cid:96) 0-1 ( F ( x ) , y )] ≤ E P [ (cid:96) β =1 ( m F ( x, y ))] ≤
[Equation]
E P n [ (cid:96) β [Equation]
1 ( m F ( x, y ))] +
[Equation]
(cid:32)
[Equation] ||| [Equation] ||| ( F
[Equation]
n +
[Equation] [Equation] n (cid:33) (C.2) Now we ﬁrst note that for a misclassiﬁed pair, (cid:96) β
=1 ( m
[Equation]
( x, y ))
= (cid:96)
[Equation]
1 ( F ( x ) , y )
= 1 .
For correctly classiﬁed examples, we also have the bound (cid:96) β
=1 ( m F ( x, y )) ≤ ( c 2 q ) q/ 2 m F ( x,y ) q for constant cindependent of q .
Thus, it follows that
[Equation]
Plugging this into (C.2), we get with probability 1 − δ for all F ∈ F ,
[Equation]
where E is deﬁned by
E = [Equation] n
[Equation]
Thus, it sufﬁces to upper bound E .
By Lemma C.1, we have m F ( x, y ) ≥ (cid:107){ κ (cid:63) i ( x, y ) /α
i } k i =1 (cid:107) − 1 p/ ( p − 1) for the choice of α, p used to deﬁne m F .
We will set p =
q/ ( q − 1) and union bound (C.3) over choices of α .
First, for a particular choice of α and p = q/ ( q − 1) , we apply our lower bound on m F ( x, y ) to simplify (C.4) as follows:
[Equation]
For convenience, we use ˜ E F ( α ) to denote (C.5) as a function of α .
Note that κ (cid:63)i depends on F .
Now let α (cid:63)F denote the minimizer of ˜ E F ( α ) .
As we do not know the exact value of α (cid:63)F before the training data is drawn, we cannot simply plug the exact value of α (cid:63)F into (C.5).
Instead, we will apply a similar union bound as the proof of Theorem A.1, although this union bound is slightly more complicated because we optimize over k quantities simultaneously.
We use ξ i to denote the lower limit on α i in our search over α , setting ξ i =
C (cid:107)·(cid:107) op ( F i ) − 1 poly ( k − 1 n − 1 ) .
Now we consider a grid of { (cid:98) α i } ki =1 , where (cid:98) α has entries of the form (cid:98)
α i = ξ i 2 j for any j ≥ 0 .
For a given choice of (cid:98) α , we assign it failure probability
[Equation]
where δ is the target failure probability after union bounding.
First, note that
[Equation]
Therefore, with probability 1 − δ , we get that (C.2) holds for m F deﬁned with respect to every (cid:98) α .
In particular, with probability 1 − δ , for all F ∈ F and (cid:98) α in the grid,
[Equation]
n
[Equation]
where the last term was obtained by subsituting (C.6) for the failure probability.
Now we claim that there is some choice of (cid:98) α in the grid such that either
[Equation]
or ˜ E F ( α (cid:63)F ) (cid:38) 1 (in which case it is trivial to obtain generalization error bounded by ˜ E F ( α (cid:63)F ) ).
To see this, we ﬁrst consider (cid:98) α in our grid such that (cid:98) α i ∈
[ α (cid:63)F,i , 2 α (cid:63)F,i + ξ i ] .
By construction of our grid of (cid:98) α , such a choice always exists.
Then we have
[Equation]
The ﬁrst term we obtained because (cid:98)
α i ≥ α
(cid:63)F,i , and the second via the upper bound (cid:98) α i ≤ 2 α
(cid:63)F,i + ξ i .
Thus, for some choice of (cid:98) α in the grid, we have ˜ E F ( (cid:98) α ) ≤ 9 ˜ E F ( α (cid:63)F ) + poly ( n − 1 ) .
Furthermore, if α (cid:63)F > c · C (cid:107)·(cid:107) op ( F i ) − 1 n for some constant c , we note that ˜ E F ( α (cid:63)F ) (cid:38) 1 - thus, it sufﬁces to only consider α (cid:63)F ≤ c · C (cid:107)·(cid:107) op ( F i ) − 1 n .
In particular, we only need to consider (cid:98)
α i where log(2 (cid:98) α
i /ξ i )
(cid:46) log kn .
Finally, we note that we can assume WLOG that k (cid:46) n otherwise (C.8) would give a trivial bound.
Combining these facts gives (C.8).
Thus, it follows that for all F ∈ F ,
[Equation]
Finally, we can apply Lemma C.2 using z i
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
and
[Table]
to get
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Substituting into (C.9) gives the desired bound.
Lemma C.2.
For coefﬁcients { z i } ki =1 , { b i } ki =1 > 0 and integer q > 0 , deﬁne
[Equation]
with minimizer α (cid:63) and minimum value E (cid:63) .
Then
[Equation]
Proof.
Choose { α i } ki =1 as follows (we obtained this by solving for α for which ∇ α E ( α ) = 0 ):
[Equation]
For this particular choice of α [Equation] we can compute
[Equation]
Likewise, we can also compute (cid:32)(cid:88)
[Equation]
4 − 4 q q ( q +2) = (cid:16) q (cid:17) 2 q +2 (cid:32)(cid:88) i ( z i b i ) 2 /(cid:33) 3 q 2+4 q − 4+4 − 4 q q ( q +2) = (cid:16) q (cid:17) 2 q
{++
 (cid:32)(cid:88) i ( z i b i ) 2 /(cid:33) 3 q q ++}
Finally, we note that (cid:0) q 2 (cid:1) 2 / ( q +2)
+ (cid:0) q 2 (cid:1) − qq +2 ≤ 2 , so we obtain
[Equation]
D L OWER B OUNDING m F FOR S MOOTH L AYERS
In this section, we prove Lemma C.1, which states that when the function F is a composition of functions with Lipschitz derivative, we will be able to lower bound m F ( x, y ) in terms of the intermediate Jacobians and layer norms evaluated at x .
To prove Lemma C.1, we rely on tools developed by (Wei and Ma, 2019) which control the change in the output of a composition of functions if all the intermediate Jacobians are bounded. First, we deﬁne the soft indicator 1 ≤ t as follows: [Equation]
We also deﬁne the ramp loss T ρ as follows: [Equation] Using the techniques of (Wei and Ma, 2019), we work with an “augmented” indicator which lower bounds the indicator that the prediction is correct, 1 [ γ ( F ( x, δ ) , y ) ≥ 0] .
We deﬁne this augmented indicator by
[Equation]
for nonnegative parameters ρ, t i , τ j ← i which we will later choose to be the margin, hidden layer norm, and Jacobian norms at the unperturbed input.
Because the augmented indicator I ( δ ; x, y ) conditions on small Jacobian and hidden layer norms, it will turn out to be κ (cid:63)i ( x, y ) -Lipschitz in the perturbation δ i . Furthermore, by construction, the value of the augmented indicator I ( δ ; x, y ) will equal 1 when δ = 0 , and we will also have
[Equation]
This immediately gives a lower bound on the perturbation level required to create a negative margin.
The lemma below formally bounds the Lipschitz constant of I ( δ ; x, y ) in δ i .
Lemma D.1.
For nonnegative parameters t i , τ j ← i , ρ , with τ j ← j +
=
1 for any j and τ j ← j (cid:48)
=
for j ≤ j (cid:48) + 2 , deﬁne the function I ( δ ; x, y ) as in (D.1) .
Then in the setting of Lemma C.1, for a given i ∈ [ k ] , for all choices of δ i and ν , if δ j = 0 for j > i , we have
[Equation]
for ˜ κ i deﬁned as follows:
[Equation]
[Equation]
i +
[Equation]
j
[Equation]
[Equation] (cid:48) [Equation] [Equation]
j
[Equation]
j
[Equation]
j
[Equation]
We prove Lemma D.1 in Section D.1.
With Lemma D.1, we can formalize the proof of Lemma C.1.
Proof of Lemma C.1. We will apply Lemma D.1, using t i = s i ( x ) , ρ = γ ( F ( x ) , y ) , τ j ← i = κ j ← i ( x ) .
First, note that for this choice of parameters, the Lipschitz constant ˜ κ i of Lemma D. evaluates to κ (cid:63)i ( x, y ) .
Thus, it follows that for all δ ,
[Equation]
Furthermore, by the deﬁnition of I ( δ ; x, y ) , we have 1 [ γ ( F ( x, δ ) , y ) ≥ 0] ≥ I ( δ ; x, y ) Finally, by our choice of the parameters used to deﬁne I ( δ ; x, y ) , we also have I (0; x, y ) ≥ 1 . Combining everything with (D.3), we get 1 [ γ ( F ( x, δ ) , y ) ≥ 0] ≥ I ( δ ; x, y ) ≥ I (0; x, y ) − (cid:88)
i κ (cid:63)i ( x, y ) (cid:107) δ i (cid:107) ≥ 1 − (cid:107){ κ (cid:63)i ( x, y ) /α i } ki =1 (cid:107) p/ ( p − 1) (cid:107){ α i (cid:107) δ i (cid:107)} ki =1 (cid:107) p (since (cid:107) · (cid:107) p/ ( p − 1) and (cid:107) · (cid:107) p are dual norms)
= 1 − (cid:107){ κ (cid:63)i ( x, y ) /α i } ki =1 (cid:107) p/ ( p − 1) ||| δ ||| Thus, for any δ , if ||| δ ||| < (cid:107){ κ (cid:63)i ( x, y ) /α
i } ki =1 (cid:107) − 1 p/ ( p − 1) , then 1 [ γ ( F ( x, δ ) , y ) ≥ 0] > 0 , which in turn implies γ ( F ( x, δ ) , y ) ≥ 0 . It follows by deﬁnition of m F ( x, y ) that m F ( x, y ) ≥ (cid:107){ κ (cid:63) i ( x, y ) /α i } k i =1 (cid:107) − 1 p/ ( p − 1) . D. P ROOF OF L EMMA D.
To see the core idea of the proof, consider differentiating I ( δ ; x, y ) with respect to δ i (ignoring for the moment that the soft indicators are technically not differentiable).
Let the terms A 1 , . . .
, A q represent the different indicators which the product I ( δ ; x, y ) is comprised of.
Then by the product rule for differentiation, we would have
[Equation]
Now the idea is that for every j , the product (cid:81)
j
[Equation]
(cid:54) =
j A j (cid:48)
( δ ; x, y ) contains an indicator that
D δ i
A j ( δ ; x, y ) is bounded – this is stated formally by Lemmas D.3, D.4, and D.5.
Informally, this allows us to bound (cid:107) D δ i
I ( δ ; x, y ) (cid:107) by the desired Lipschitz constant ˜ κ i .
To formally prove this statement for the case of non-differentiable functions (as the soft-indicators≤ t are non-differentiable), it will be convenient to introduce the following notion of productLipschitzness: for functions
A 1 : D I
→ R + and A 2 : D I
→ R + , where D I is some normed space, we say that function A 1 is ¯ τ -product-Lipschitz w.r.t.
A 2 if there exists some c, C > 0 such that for any (cid:107) ν (cid:107) ≤ c and x ∈ D
I , we have
[Equation]
We use the following fact that the product of functions which are product-Lipschitz with respect to one another is in fact Lipschitz.
We provide the proof in Section D.2.
Lemma D.2.
Let A 1 , . . .
, A q : D I → [0 , 1] be a set of Lipschitz functions such that A
i is ¯
τ i -productLipschitz w.r.t (cid:81)
j (cid:54) =
i
A j for all i .
Then the product (cid:81) i
A i is 2 (cid:80)
i ¯ τ i -Lipschitz.
Now we proceed to formalize the intuition of product-rule differentiation presented above, by showing that the individual terms in I ( δ ; x, y ) are product-Lipschitz with respect to the other terms.
For the following three lemmas, we require the technical assumption that for any ﬁxed choice of x, δ − i , the functions f
j ← 1 ( x, δ ) , J j (cid:48) ← j (cid:48)(cid:48) ( x, δ ) are worst-case Lipschitz in δ i as measured in (cid:107) · (cid:107) , (cid:107) · (cid:107) op , respectively, with Lipschitz constant C (cid:48) .
Our proof of Lemma D.1, however, can easily circumvent this assumption.
The proofs of the following three lemmas are given in Section D.2.
Lemma D.3.
Choose i, j with k − 1 ≥ j ≥ i .
Then after we ﬁx any choice of x, δ − i , the function 1 ≤ t j ( (cid:107) f
j ← 1 ( x, δ ) (cid:107) ) is 4 τ j ← i
+
t
i − 1 t j -product-Lipschitz in δ i with respect to 1 ≤ τ j ← i +1 ( (cid:107) J j ← i +1 ( x, δ ) (cid:107) op ) 1 ≤ t i − 1 ( (cid:107) f
i − 1 ← 1 ( x, δ ) (cid:107) ) .
Lemma D.4.
Choose i ≤ k .
Then after we ﬁx any choice of x, δ − i , the function
[Equation]
is
[Equation]
-product-Lipschitz in δ i with respect to
[Equation]
Lemma D.5.
Choose i, j 1 , j 2 with j 1 ≥ j 2 , j 1 > i .
Set product-Lipschitz constant ¯ τ as follows:
[Equation]
τ [Equation] Then for any ﬁxed choice of x, δ − i satisfying δ j = 0 for j > i , the function
[Equation]
is ¯ τ -product-Lipschitz in δ i with respect to
[Equation]
Here note that we have τ
[Equation]
if
[Equation]
− [Equation] ≤
[Equation] Given the described steps, we will now complete the proof of Lemma D.1.
Proof of Lemma D.1.
We ﬁrst assume that the conditions of Lemmas D.3, D.4, D.
regarding C (cid:48) - worst-case Lipschitzness hold.
We note that I ( δ ; x, y ) is a product which contains all the functions appearing in Lemmas D.3, D.4, and D.5.
Thus, Claim D.1 allows us to conclude that each term in
I ( δ ; x, y ) is product-Lipschitz with respect to the product of the remaining terms.
As these lemmas also account for all the terms in I ( δ ; x, y ) , we can thus apply Lemma D.2, where each A i is set to be a term in the product for I ( δ ; x, y ) .
Therefore, to bound the Lipschitz constant in δ i of I ( δ ; x, y ) , we sum the product-Lipschitz constants given by Lemmas D.3, D.4, and D.5.
This gives that I ( δ ; x, y ) is ˜ κ i -Lipschitz in δ i for ˜ κ i deﬁned in (D.2).
Now to remove the C (cid:48) worst-case Lipschitzness assumption, we can follow the reasoning of Claim
D.6 of (Wei and Ma, 2019) to note that such Lipschitz constants exist if we restrict δ i to some compact set, and thus conclude the lemma statement for δ i restricted to this compact set.
Now we simply choose this compact set sufﬁciently large to include both δ i and δ i
+ ν .
D. P ROOFS FOR P RODUCT -L IPSCHITZ L EMMAS
Proof of Lemma D.2.
As each A i is Lipschitz and there are a ﬁnite number of functions, there exists
C (cid:48) such that any possible product
A
i 1 A i 2 · · · A i j is C (cid:48)
-Lipschitz.
Furthermore, by the deﬁnition of product-Lipschitz, there are c, C > 0 such that for any (cid:107) ν (cid:107) ≤ c , x ∈ D
I , and 1 ≤ i ≤ q , we have
[Equation]
Now we note that
[Equation]
Now for any i , we have
[Equation]
i ( x )) | ( i − 1 (cid:89) j =
A j ( x ) + C (cid:48) (cid:107) ν (cid:107) ) q
(cid:89)
j = i +
{+A j ( x ) (as (cid:81) i − 1 j =
 A j is C (cid:48) -Lipschitz) ≤ | A i ( x + ν ) − A+}
i ( x ) | ( (cid:89) j (cid:54) = i
A j ( x ) + C (cid:48) (cid:107) ν (cid:107) )
We used the fact that (cid:81) qj
= i +
A j ( x ) ≤ 1 .
Now we have |
A i ( x + ν ) − A i ( x ) | (cid:81) j (cid:54) = i
A j ( x ) ≤ ¯ τ i (cid:107)
ν (cid:107) + C (cid:107) ν
(cid:107) 2 , and | A i ( x + ν ) − A i ( x ) | C (cid:48) (cid:107) ν (cid:107) ≤ C (cid:48) 2 (cid:107) ν (cid:107) 2 as A i is C (cid:48)
-Lipschitz, so | ( A i ( x + ν )
− A i ( x ))
i − 1 (cid:89) j =
A j ( x + ν ) q (cid:89) j
= i +
A j ( x ) | ≤ ¯ τ i (cid:107) ν (cid:107) + ( C + C (cid:48) 2 ) (cid:107) ν (cid:107)
Plugging this back into (D.4) and applying triangle inequality, we get | (cid:89)
i
[Equation]
(cid:33)
Deﬁne the constant C (cid:48)(cid:48) (cid:44) min { c, (cid:80) i
¯ τ i q ( C + C (cid:48) 2 ) } .
For any x and all ν satisfying (cid:107) ν (cid:107) ≤ C (cid:48)(cid:48) , we have | (cid:89)
i
A i ( x + ν ) − (cid:89) i A i ( x ) | ≤ 2 (cid:107) ν (cid:107) (cid:88)
i ¯ τ i (D.5)
Now for any x, y ∈
D I , we wish to show | (cid:81)
i A i
( y ) − (cid:81)
i
A i ( x ) | ≤ 2 (cid:107) x − y (cid:107) (cid:80)
i
¯ τ i .
To this end, we divide x − y into segments of length at most C (cid:48)(cid:48) and apply (D.5) on each segment.
Deﬁne x ( j ) =
x
+ j C (cid:48)(cid:48) (cid:107) x − y (cid:107) ( y − x ) for j = 1 , . . .
, (cid:98)(cid:107) x − y (cid:107) /C
(cid:48)(cid:48) (cid:99) .
Then as (cid:107) x
( j ) − x ( j − 1) (cid:107) ≤ C (cid:48)(cid:48) , we have | (cid:81) i
A i ( x ( j ) ) − (cid:81)
i
A i ( x ( j − 1) )
| ≤ (cid:107) x
( j ) − x ( j − 1) (cid:107) (cid:80)
i ¯ τ i .
Furthermore, we note that the sum of all the segment lengths equals (cid:107) y − x (cid:107) .
Thus, we can sum this inequality over pairs ( x, x (1) ) , . . .
, ( x ( (cid:98)(cid:107) x − y [Equation]
) , y ) and apply triangle inequality to get
[Equation]
as desired.
Proof of Lemma D.3.
For convenience, deﬁne
[Equation]
We ﬁrst note that D δ i f
j ← 1 ( x, δ ) , the partial derivative of f j ← 1 with respect to δ i , is given by
J j ← i +1 ( x, δ ) [Equation]
f i − 1 ← 1 ( x, δ ) (cid:107) by Claim D.2.
As J j ← i +1 ( x, δ ) , (cid:107) f
i − 1 ← 1 ( x, δ ) (cid:107) are both worst-case
Lipschitz in δ i with some Lipschitz constant C (cid:48) , we can apply Claim H.4 of (Wei and Ma, 2019) to obtain:
[Equation]
for any ν and some Lipschitz constant C (cid:48)(cid:48) .
Thus, by the t − 1 j
Lipschitz-ness of the indicator 1 ≤ t j , we have
[Equation]
t j Now by deﬁnition of A ( x, δ ) , we get that the right hand side equals 0 if (cid:107) J j ← i +1 ( x, δ ) (cid:107) op ≥ 2 τ
j ← i +1 or (cid:107) f
i − 1 ← 1 ( x, δ ) (cid:107) ≥ 2 t i − 1 .
Thus, the right hand side must be bounded by
[Equation]
which gives product-Lipschitzness with constant
[Equation]
j ←
[Equation]
+
[Equation]
1 t j .
Proof of Lemma D.4.
This proof follows in an identical manner to that of Lemma D.3.
The only additional step is using the fact that γ ( h, y ) is 1-Lipschitz in h , so the composition T ρ ( γ ( h, y )) is ρ − 1 -Lipschitz in h .
Proof of Lemma D.5.
Let C (cid:48) be an upper bound on the Lipschitz constant in δ i of all the functions
J j (cid:48) ← j (cid:48)(cid:48)
( x, δ ) .
As we assumed that each f j has κ (cid:48)
j -Lipschitz Jacobian, such an upper bound exists.
We ﬁrst argue that
[Equation]
for some Lipschitz constant C (cid:48)(cid:48) .
The proof of this statement is nearly identical to the proof of Claim
D.3 in (Wei and Ma, 2019)
[Equation]
so we only sketch it here and point out the differences.
We rely on the expansion
[Equation]
which follows from the chain rule.
Now we note that we can compute the change in a single term
J
[Equation]
( x, δ ) from perturbing δ i as follows:
[Equation]
Note that when j (cid:48) > i , by assumption δ j (cid:48)
= 0 , so D h f j (cid:48)
← j (cid:48) ( h, δ )
=
D h f j (cid:48)
( h ) .
Thus, as f j (cid:48) has κ (cid:48)
j (cid:48) -Lipschitz derivative, we get
[Equation]
(since the derivative of f j (cid:48) is κ (cid:48)
j (cid:48) -Lipschitz)
[Equation]
(by Claim H.4 of (Wei and Ma, 2019))
[Equation]
We obtained the last line via Claim D.2.
We note that the cases when j (cid:48) >
i contribute to the terms under the summation in (D.6).
When j (cid:48)
=
i , we have D
h
f
i ← i ( h, δ ) =
D
h
f
i
( h ) + D h
[ δ i (cid:107) h (cid:107) ] for any h , so (cid:107) D
h f i ← i ( h, δ i , δ − i ) −
D h
f i ← i ( h, δ i + ν, δ − i ) (cid:107) op
= (cid:107) D h [ ν (cid:107) h (cid:107) ] (cid:107) op ≤ (cid:107) ν
(cid:107) .
As this holds for any h , it follows that (cid:107)
[Equation]
[Equation]
) (cid:107) [Equation] (cid:107) [Equation]
(cid:107) .
This term results in the last quantity in (D.6).
Finally, when j (cid:48) <
i , we have J j (cid:48)
← j (cid:48) ( x, δ − i , δ i + ν ) =
J j (cid:48) ← j (cid:48)
( x, δ − i , δ i ) as J j (cid:48)
← j (cid:48) does not depend on δ i .
To see how (D.6) follows, we would apply the above bounds in a telescoping sum over indices j (cid:48) ranging from max { j 2 , i } to j 1 .
For a more detailed derivation, refer to the steps in Claim D.3 of (Wei and Ma, 2019).
Now for convenience deﬁne
[Equation]
Note that if any of the bounds set by the indicators in A ( x, δ ) are violated, then A ( x, δ ) = 0 , and thus
[Equation]
In the other case, we have (cid:107) f i − 1 ← 1 ( x, δ ) (cid:107) ≤ 2 t
i − 1 , and (cid:107) J j
[Equation]
← j
[Equation]
( x, δ )
[Equation]
op ≤ 2 τ j
[Equation]
←
[Equation]
, in which case (D.6) can be bounded by
[Equation]
+
[Equation]
for some C (cid:48)(cid:48)(cid:48) that is independent of x, δ, ν .
Thus, by Lipschitz-ness of 1 ≤ τ j 1 ← j 2 ( · ) and the triangle inequality, we have
[Equation]
This gives the desired result.
Claim D.1.
Let A 1 , A 2 , A 3 : D I → [0 , 1] be functions where A 1 is ¯ τ -product-Lipschitz w.r.t.
A 2 .
Then A 1 is also ¯ τ -product Lipschitz w.r.t.
A 2 A 3 .
Proof.
This statement follows from the deﬁnition of product-Lipschitzness and the fact that
[Equation]
since A 3 ( x ) ≤ 1 .
Claim D.2.
The partial derivative of f j ← 1 with respect to variable δ i evaluated at x, δ can be computed as
[Equation]
Proof.
By deﬁnition, f j ← 1 ( x, δ ) =
f
j ← i {++
+} ( f i [Equation] 1 ( x, δ ) , δ ) .
Thus, we note that f j ← 1 ( x, δ ) only depends on δ i through f i ← 1 ( x, δ ) , so by chain rule we have
[Equation]
In the second line [Equation] we invoked the deﬁnition of J j ← i +1 ( x, δ ) .
Claim D.3.
We have the expansion
[Equation]
Proof.
This is a result of the chain rule [Equation] but for completeness we state the proof here.
We have
[Equation]
Thus, plugging in h = f i − 1 ← 1 ( x, δ ) , we get
[Equation]
Now we can apply identical steps to expand J j − 1 ← i ( x, δ ) , giving the desired result.
E P ROOFS FOR A DVERSARIALLY R OBUST C LASSIFICATION
In this section, we derive the generalization bounds for adversarial classiﬁcation presented in Section 4.
Recall the adversarial all-layer margin m adv F ( x, y ) (cid:44) min x (cid:48) ∈B adv ( x ) m F ( x (cid:48) , y ) deﬁned in Section 4.
In this section, we will use the general deﬁnition of m F in (A.1).
We will sketch the proof of Theorem E.1 using the same steps as those laid out in Sections 2 and A.
We will rely on the following general analogue of Theorem C.1 with the exact same proof, but with κ (cid:63)i (deﬁned in Section C) replaced by κ adv i ( x, y ) (cid:44) max x (cid:48) ∈B
adv ( x ) κ (cid:63)i ( x (cid:48) , y ) everywhere:
Theorem E.1.
Let F = { f k ◦ · · · ◦ f 1 : f i ∈ F i } denote a class of compositions of functions from k families {F i } ki =1 , each of which satisﬁes Condition A.1 with operator norm (cid:107) · (cid:107) op and complexity
C (cid:107)·(cid:107) op ( F i ) .
For any choice of integer q > 0 , with probability 1 − δ for all F ∈ F the following bound holds: [Equation] F [Equation]
x
[Equation]
y )
] ≤
[Equation]
where S adv n denotes the subset of training examples correctly classiﬁed by F with respect to adversarial perturbations.
In particular, if F classiﬁes all training samples with adversarial error 0, i.e.
|S adv n | = n , with probability 1 − δ we have
[Equation]
ζ where ζ (cid:44) O (cid:16) k log n +log(1 /δ )
n (cid:17) is a low order term.
Given Theorem E.1, Theorem 4.1 follows with the same proof as the proof of Theorem 3.1 given in
Section B. To prove Theorem E.1, we ﬁrst have the following analogue of Lemma A.1 bounding the covering number of m adv ◦ F .
Lemma E.1.
Deﬁne m adv ◦ F (cid:44) { ( x, y ) (cid:55)→ m adv F ( x, y ) : F ∈ F} .
Then
[Equation]
[Equation] [Equation] This lemma allows us to invoke Lemma A.2 on a smooth loss composed with m adv ◦ F , as we did for the clean classiﬁcation setting.
The lemma is proven the exact same way as Lemma A.1, given the Lipschitz-ness of m adv F below:
Claim E.1.
For any x, y ∈ D 0 ×
[ l ] , and function sequences F = { f i } ki =1 , (cid:98) F = { (cid:98) f i } ki =1 , we have [Equation]
Proof.
Let x (cid:63) ∈ B
adv ( x ) be such that m adv F ( x, y )
=
m F ( x (cid:63) , y ) .
By Claim A.1, we have
[Equation]
We can apply the reverse reasoning to also obtain m adv
[Equation]
x,
[Equation]
≤ m
[Equation]
Combining the two gives us the desired result.
Next, we lower bound m adv F when each function in F is smooth.
Lemma E.2.
In the setting of Lemma C.1, let κ adv i
( x, y ) (cid:44) max x (cid:48) ∈B adv ( x ) κ (cid:63)i ( x (cid:48) , y ) .
Then if (cid:96) adv0-1 ( F ( x ) , y ) = 0 , we have
[Equation]
Proof.
By deﬁnition and Lemma C.1, we have
[Equation]
To ﬁnish the proof of Theorem E.1, we use (cid:96) β
=1 ( m adv F ( x, y )) as an upper bound for (cid:96) adv0-1 ( F ( x ) , y ) and follow the steps of Theorem C.1 to optimize over the choice of { α i } ki =1 .
As these steps are identical to Theorem C.1, we omit them here.
With Theorem E.1 in hand, we can conclude Theorem 4. using the same proof as Theorem 3.1.
F A DDITIONAL E XPERIMENTAL D ETAILS F. A DDITIONAL D ETAILS FOR C LEAN C LASSIFICATION S ETTING
For the experiments presented in Table 1, the other hyperparameters besides t and η perturb are set to their defaults for WideResNet architectures.
Our code is inspired by the following PyTorch WideResNet implementation: https://github.com/xternalz/
WideResNet-pytorch , and we use the default hyperparameters in this implementation.
Although we tried a larger choice of t , the number of updates to the perturbations δ , our results did not depend much on t .
[Table]
In Table 3, we demonstrate that our AMO algorithm can also improve performance for conventional feedforward architectures such as VGG (Simonyan and Zisserman, 2014).
We report the best validation error on the clean classiﬁcation setting.
For both methods, we train VGG-19 for 350 epochs using weight decay 0 . 0005 and an initial learning rate of 0.05 annealing by a factor of 0.1 at the 150-th and 250-th epochs.
For the AMO algorithm, we optimize perturbations δ for t = 1 steps and use learning rate η perturb = 0 . 01 .
F. A DDITIONAL D ETAILS FOR R OBUST AMO
For both methods, we produce the perturbations during training using 10 steps of PGD with (cid:96) ∞ perturbations with radius (cid:15) = 8 . For all architectures and methods we use a weight decay of 0 .and
initial learning rate of 0 . 1 annealed by a factor of 0.1 at the 100-th and 150-th epochs.
We train for 200 epochs total with a batch size of 128.
We update the adversarially perturbed training inputs with a step size of 2 in pixel space.
For the models trained using robust AMO, the learning rate on perturbations δ is set to 0.0008 and the (cid:96) 2 -regularization on δ is set to 100 (resulting in a constant factor shrinkage in δ over 10 steps of PGD).
For the WideResNet-16-10 model trained using robust AMO, we apply perturbations following every convolutional layer in the residual blocks, whereas for the WideResNet-28-10 model, we apply the perturbations only after the ﬁrst convolution in each residual block because of GPU memory constraints.