O N THE M ARGIN T HEORY OF F EEDFORWARD N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Past works have shown that, somewhat surprisingly, over-parametrization can help generalization in neural networks.
Towards explaining this phenomenon, we adopt a margin-based perspective.
We establish: 1) for multi-layer feedforward relu networks, the global minimizer of a weakly-regularized cross-entropy loss has the maximum normalized margin among all networks, 2) as a result, increasing the over-parametrization improves the normalized margin and generalization error bounds for deep networks.
In the case of two-layer networks, an inﬁnite-width neural network enjoys the best generalization guarantees.
The typical inﬁnite feature methods are kernel methods; we compare the neural net margin with that of kernel methods and construct natural instances where kernel methods have much weaker generalization guarantees.
We validate this gap between the two approaches empirically.
Finally, this inﬁnite-neuron viewpoint is also fruitful for analyzing optimization.
We show that a perturbed gradient ﬂow on inﬁnite-size networks ﬁnds a global optimizer in polynomial time.
I NTRODUCTION
In deep learning, over-parametrization refers to the widely-adopted technique of using more parameters than necessary (Krizhevsky et al., 2012;
Livni et al., 2014).
Both computationally and statistically, over-parametrization is crucial for learning neural nets.
Controlled experiments demonstrate that over-parametrization eases optimization by smoothing the non-convex loss surface (Livni et al., 2014;
Sagun et al., 2017).
Statistically, increasing model size without any regularization still improves generalization even after the model interpolates the data perfectly (Neyshabur et al., 2017b).
This is surprising given the conventional wisdom on the trade-off between model capacity and generalization.
In the absence of an explicit regularizer, algorithmic regularization is likely the key contributor to good generalization.
Recent works have shown that gradient descent ﬁnds the minimum norm solution ﬁtting the data for problems including logistic regression, linearized neural networks, and matrix factorization (Soudry et al., 2018; Gunasekar et al., 2018b;
Li et al., 2018; Gunasekar et al., 2018a; Ji & Telgarsky, 2018).
Many of these proofs require a delicate analysis of the algorithm’s dynamics, and some are not fully rigorous due to assumptions on the iterates.
To the best of our knowledge, it is an open question to prove analogous results for even two-layer relu networks.
(For example, the technique of Li et al.
(2018) on two-layer neural nets with quadratic activations still falls within the realm of linear algebraic tools, which apparently do not sufﬁce for other activations.)
We propose a different route towards understanding generalization: making the regularization explicit .
The motivations are: 1) with an explicit regularizer, we can analyze generalization without fully understanding optimization; 2) it is unknown whether gradient descent provides additional implicit regularization beyond what (cid:96) 2 regularization already offers; 3) on the other hand, with a sufﬁciently weak (cid:96) 2 regularizer, we can prove stronger results that apply to multi-layer relu networks.
Additionally, explicit regularization is perhaps more relevant because (cid:96) 2 regularization is typically used in practice.
Concretely, we add a norm-based regularizer to the cross entropy loss of a multi-layer feedforward neural network with relu activations.
We show that the global minimizer of the regularized objective achieves the maximum normalized margin among all the models with the same architecture, if the regularizer is sufﬁciently weak (Theorem 2.1).
Informally, for models with norm 1 that perfectly classify the data, the margin is the smallest difference across all datapoints between the classiﬁer score for the true label and the next best score.
We are interested in normalized margin because its inverse bounds the generalization error (see recent work (Bartlett et al., 2017; Neyshabur et al., 2017a; 2018; Golowich et al., 2017) or Proposition 3.1).
Our work explains why optimizing the training loss can lead to parameters with a large margin and thus, better generalization error (see Corollary 3.2).
We further note that the maximum possible margin is non-decreasing in the width of the architecture, and therefore the generalization bound of Corollary 3.2 can only improve as the size of the network grows (see Theorem 3.3).
Thus, even if the dataset is already separable, it could still be useful to increase the width to achieve larger margin and better generalization.
At a ﬁrst glance, it might seem counterintuitive that decreasing the regularizer is the right approach.
At a high level, we show that the regularizer only serves as a tiebreaker to steer the model towards choosing the largest normalized margin.
Our proofs are simple, oblivious to the optimization procedure, and apply to any norm-based regularizer.
We also show that an exact global minimum is unnecessary: if we approximate the minimum loss within a constant factor, we obtain the max-margin within a constant factor (Theorem 2.2).
To better understand the neural network max-margin, in Section 4 we compare the max-margin two-layer network obtained by optimizing both layers jointly to kernel methods corresponding to ﬁxing random weights for the hidden layer and solving a 2-norm max-margin on the top layer.
We design a simple data distribution (Figure 1) where neural net margin is large but the kernel margin is small.
This translates to an Ω( √ d ) factor gap between the generalization error bounds for the two approaches and demonstrates the power of neural nets compared to kernel methods.
We experimentally conﬁrm that a gap does indeed exist.
In the setting of two-layer networks, we also study how over-parametrization helps optimization.
Prior works (Mei et al., 2018; Chizat & Bach, 2018; Sirignano & Spiliopoulos, 2018; Rotskoff & Vanden-Eijnden, 2018) show that gradient descent on two-layer networks becomes Wasserstein gradient ﬂow over parameter distributions in the limit of inﬁnite neurons.
For this setting, we prove that perturbed Wasserstein gradient ﬂow ﬁnds a global optimizer in polynomial time.
Finally, we empirically validate several claims made in this paper.
First, we conﬁrm that neural networks do generalize better than kernel methods. Second, we show that for two-layer networks, the test error decreases and margin increases as the hidden layer grows, as predicted by our theory.
1. A DDITIONAL R ELATED W ORK
Zhang et al.
(2016) and Neyshabur et al.
(2017b) show that neural network generalization deﬁes conventional explanations and requires new ones.
Neyshabur et al. (2014) initiate the search for the “inductive bias” of neural networks towards solutions with good generalization.
Recent papers (Hardt et al., 2015; Brutzkus et al., 2017; Chaudhari et al., 2016) study inductive bias through training time and sharpness of local minima.
Neyshabur et al.
(2015a) propose a new steepest descent algorithm in a geometry invariant to weight rescaling and show that this improves generalization.
Morcos et al.
(2018) relate generalization in deep nets to the number of “directions” in the neurons.
Other papers (Gunasekar et al., 2017; Soudry et al., 2018; Nacson et al., 2018; Gunasekar et al., 2018b; Li et al., 2018; Gunasekar et al., 2018a) study implicit regularization towards a speciﬁc solution.
Ma et al.
(2017) show that implicit regularization can help gradient descent avoid overshooting optima.
Rosset et al.
(2004a;b) study logistic regression with a weak regularization and show convergence to the max margin solution.
We adopt their techniques and extend their results.
A line of work initiated by Neyshabur et al. (2015b) has focused on deriving tighter norm-based Rademacher complexity bounds for deep neural networks (Bartlett et al., 2017; Neyshabur et al., 2017a; Golowich et al., 2017) and new compression based generalization properties (Arora et al., 2018b).
Dziugaite & Roy (2017) manage to compute non-vacuous generalization bounds from PAC-Bayes bounds.
Neyshabur et al.
(2018) investigate the Rademacher complexity of two-layer networks and propose a bound that is decreasing with the distance to initialization.
Liang & Rakhlin (2018) and Belkin et al.
(2018) study the generalization of kernel methods.
On the optimization side, Soudry & Carmon (2016) explain why over-parametrization can remove bad local minima.
Safran & Shamir (2016) show that over-parametrization can improve the quality of the random initialization.
Haeffele & Vidal (2015), Nguyen & Hein (2017), and Venturi et al.
(2018) show that for sufﬁciently overparametrized networks, all local minima are global, but do not show how to ﬁnd these minima via gradient descent.
Du & Lee (2018) show that for two-layer networks with quadratic activations, all second-order stationary points are global minimizers.
Arora et al.
(2018a) interpret over-parametrization as a means of implicit acceleration during optimization.
Mei et al.
(2018), Chizat & Bach (2018), and Sirignano & Spiliopoulos (2018) take a distributional view of over-parametrized networks.
Chizat & Bach (2018) show that Wasserstein gradient ﬂow converges to global optimizers under structural assumptions.
We extend this to a polynomial-time result.
1. N OTATION
Let R denote the set of real numbers.
We will use (cid:107)·(cid:107) to indicate a general norm, with (cid:107)·(cid:107) 1 , (cid:107)·(cid:107) 2 , (cid:107)·(cid:107) ∞
denoting the (cid:96) 1 , (cid:96) 2 , (cid:96) ∞ norms on ﬁnite dimensional vectors, respectively, and (cid:107) · (cid:107) F denoting the
Frobenius norm on a matrix.
In general, we use ¯ on top of a symbol to denote a unit vector: when applicable, ¯ u (cid:44) u/ (cid:107) u (cid:107) , where the norm (cid:107) · (cid:107) will be clear from context.
Let S d − 1 (cid:44) { ¯ u ∈ R d : (cid:107) ¯ u (cid:107) 2 = 1 } be the unit sphere in d dimensions.
Let L p ( S d − 1 ) be the space of functions on S d −for which the p -th power of the absolute value is Lebesgue integrable.
For α ∈ L p ( S d − 1 ) , we overload notation and write (cid:107) α (cid:107) p (cid:44) (cid:0)(cid:82) S d − 1 | α (¯ u )
| p
d ¯ u (cid:1) 1 /p .
Additionally, for α 1 ∈ L 1 ( S d − 1 ) and α 2 ∈ L ∞ ( S d − 1 ) or α 1 , α 2 ∈
L 2 ( S d − 1 ) , we can deﬁne (cid:104)
α 1 , α 2 (cid:105) (cid:44) (cid:82) S
d − 1 α 1 (¯ u ) α 2 (¯ u ) d ¯ u < ∞ .
Furthermore, we will use Vol ( S d − 1 ) (cid:44) (cid:82) S
d − 1 1 d ¯ u .
Throughout this paper, we reserve the symbol
X =
[ x 1 , . . .
, x n ] to denote the collection of datapoints (as a matrix), and Y =
[ y 1 , . . .
, y n ] to denote labels.
We use d to denote the dimension of our data.
We often use Θ to denote the parameters of a prediction function f , and f (Θ; x ) to denote the prediction of f on datapoint x .
We will use the notation (cid:46) , (cid:38) to mean less than or greater than up to a universal constant, respectively.
Unless stated otherwise, O ( · ) , Ω( · ) denote some universal constant in upper and lower bounds, respectively.
The notation poly denotes a universal constant-degree polynomial in the arguments.
W EAK R EGULARIZER G UARANTEES M AX M ARGIN S OLUTIONS
In this section, we will show that when we add a weak regularizer to cross-entropy loss with a positive-homogeneous prediction function, the normalized margin of the optimum converges to some max-margin solution.
As a concrete example, feedforward relu networks are positive-homogeneous.
Let l be the number of labels, so the i -th example has label y i ∈
[ l ] .
We work with a family F of prediction functions f (Θ; · ) : R d → R l that are a -positive-homogeneous in their parameters for some a > 0 : f ( c Θ; x ) =
c a f (Θ; x ) , ∀ c > 0 .
We additionally require that f is continuous in Θ .
For some general norm (cid:107) · (cid:107) , we study the λ -regularized cross-entropy loss L λ , deﬁned as
[Equation]
for ﬁxed r > 0 .
Let
Θ λ ∈ arg min
L λ (Θ) .
1 We deﬁne the normalized margin of Θ λ as:
[Equation]
Deﬁne the (cid:107) · (cid:107) -max normalized margin as
[Equation]
and let Θ (cid:63) be a parameter achieving this maximum.
We show that with sufﬁciently small regularization level λ , the normalized margin γ λ approaches the maximum margin γ (cid:63) .
Our theorem and proof are inspired by the result of Rosset et al.
(2004a;b), who analyze the special case when f is a linear predictor.
In contrast, our result can be applied to non-linear f as long as f is homogeneous.
Theorem 2.1.
Assume the training data is separable by a network f (Θ (cid:63) ; · ) ∈ F with an optimal normalized margin γ (cid:63) > 0 .
Then, the normalized margin of the global optimum of the weaklyregularized objective (equation 2.1) converges to γ (cid:63) as the strength of the regularizer goes to zero.
Mathematically, let γ λ be deﬁned in equation 2.2.
Then
[Equation]
An intuitive explanation for our result is as follows: because of the homogeneity, the loss L (Θ λ ) roughly satisﬁes the following (for small λ , and ignoring problem parameters such as n ):
[Equation]
Thus, the loss selects parameters with larger margin, while the regularization favors parameters with a smaller norm.
The full proof of the theorem is deferred to Section A.1.
Theorem 2.1 applies to feedforward relu networks and states that global minimizers of the weaklyregularized loss will obtain a maximum margin among all networks of the given architecture.
By considering global minimizers, Theorem 2.1 provides a framework for directly analyzing generalization properties of the solution without considering details of the optimization algorithm.
In Sectionwe leverage this framework and existing generalization bounds (Golowich et al., 2017) to provide a clean argument that over-parameterization can improve generalization.
We can also provide an analogue of Theorem 2.1 for the binary classiﬁcation setting.
For this setting, our prediction is now a single real output and we train using logistic loss.
We provide formal deﬁnitions and results in Section A.2.
Our study of the generalization properties of the max-margin (see Section 3 and Section 4) is based in this setting.
2. O PTIMIZATION A CCURACY
Since L λ is typically hard to optimize exactly for neural nets, we study how accurately we need to optimize L λ to obtain a margin that approximates γ (cid:63) up to a constant.
The following theorem shows that it sufﬁces to ﬁnd Θ (cid:48) achieving a constant factor multiplicative approximation of L λ (Θ λ ) , where λ is some sufﬁciently small polynomial in n, l, γ (cid:63) .
Though our theorem is stated for the general multi-class setting, it also applies for binary classiﬁcation.
We provide the proof in Section A.3.
Theorem 2.2.
In the setting of Theorem 2.1, suppose that we choose
[Equation]
for sufﬁciently large c (that only depends on r/a ).
For β ≤ 2 , let Θ (cid:48) denote a β -approximate minimizer of L λ , so L λ (Θ (cid:48) ) ≤ βL λ (Θ λ ) .
Denote the normalized margin of Θ (cid:48) by γ (cid:48) .
Then γ (cid:48) [Equation] 10 · β a/ [Equation]
G ENERALIZATION PROPERTIES OF A MAXIMUM MARGIN NEURAL NETWORK
In Section 2 we showed that optimizing a weakly-regularized logistic loss leads to the maximum normalized margin.
We now study the direct implications of this result on the generalization properties of the solution.
Speciﬁcally, we use existing Rademacher complexity bounds of Golowich et al. (2017) to present a generalization bound that depends on the network architecture only through the inverse (cid:96)
2 -normalized margin and depth of the network (see Proposition 3.1).
Next, we combine this bound with Theorem 2.1 to conclude that parameters obtained by optimizing logistic loss with weak (cid:96) 2 -regularization will have a generalization bound that scales with the inverse of the maximum possible margin and depth. Finally, we note that the maximum possible margin can only increase as the size of the network grows, which suggests that increasing the size of the network improves the generalization of the solution (see Theorem 3.3).
We consider depth- K neural networks with 1-Lipschitz, 1-positive-homogeneous activation φ for
K ≥ 2 .
Suppose that the collection of parameters Θ is given by matrices W 1 , . . , W K . The K -layer network will compute a real-valued score
[Equation]
where we overload notation to let φ ( · ) denote the element-wise application of the activation φ . Let m i denote the size of the i -th hidden layer, so W 1 ∈ R m 1 × d , W 2 ∈ R m 2 × m 1 , · · · , W K ∈ R 1 × m K − 1 .
We will let M (cid:44) ( m 1 , . . .
, m K − 1 ) denote the sequence of hidden layer sizes.
We will focus on (cid:96) 2 -regularized loss.
The weakly-regularized logistic loss of the depth- K architecture with hidden layer sizes M is therefore
[Equation]
We note that f is K -homogeneous in Θ , so the results of Section 2 apply to L λ, M . {+
+}
Following our conventions from Section 2, we denote the optimizer of L λ, M by Θ λ, M , the normalized margin of Θ λ, M by γ λ, M , the max-margin solution by Θ (cid:63), M , and the max-margin by γ (cid:63), M .
Our notation emphasizes the architecture of the network.
Since the classiﬁer f now predicts a single real value, we need to redeﬁne
[Equation]
When the data is not separable by a neural network with architecture M , we deﬁne γ (cid:63), M to be zero.
Recall that X
=
[ x 1 , . . .
, x n ] denotes the matrix with all the data points as columns, and Y =
[ y 1 , . . .
, y n ] denotes the labels.
We sample X and Y i.i.d.
from the data generating distribution p data , which is supported on X × {− 1 , +1 } .
We can deﬁne the population 0-1 loss and training 0-1 loss of the network parametrized by Θ by
[Equation]
Let C (cid:44) sup x ∈X (cid:107) x (cid:107) 2 be an upper bound on the norm of a single datapoint.
Proposition 3.1 shows that the generalization error only depends on the parameters through the inverse of the margin on the training data.
We obtain Proposition 3.1 by applying Theorem 1 of Golowich et al. (2017) with the standard technique of using margin loss to bound classiﬁcation error. There exist other generalization bounds which depend on the margin and some normalization (Neyshabur et al., 2015b; 2017a; Bartlett et al., 2017; Neyshabur et al., 2018); we choose the bounds of Golowich et al. (2017) because they ﬁt well with (cid:96) 2 normalization.
In the two-layer case K = 2 , the bound below also follows from
Neyshabur et al.
(2015b).
Proposition 3.1.
[Straightforward consequence of Golowich et al.
(2017, Theorem 1)] Suppose φ is 1-Lipschitz and 1 -positive-homogeneous.
For any depth- K network f (Θ; · ) separating the data with normalized margin γ (cid:44) min i y i f (¯Θ; x i ) > 0 , with probability at least 1 − δ over the draw of X, Y ,
[Equation]
where (cid:15) ( γ ) (cid:44) (cid:113) loglog 2 4 C γ n + (cid:113) log(1 /δ )
n .
Note that (cid:15) ( γ ) is typically small, and thus the above bound mainly scales with
[Equation]
For completeness, we state the proof in Section C.1. By combining this bound with our Theorem 2.
we can conclude that optimizing weakly-regularized logistic loss gives us generalization error bounds that depend on the maximum possible margin of a network with the given architecture. Corollary 3.2.
In the setting of Proposition 3.1, with probability 1 − δ ,
[Equation]
where (cid:15) ( γ ) is deﬁned as in Proposition 3.1.
Above we implicitly assume γ (cid:63), M > 0 , since otherwise the right hand side of the bound is vacuous.
By applying Theorem 2.2 with Proposition 3.1, we can also conclude that optimizing L λ, M within a constant factor gives a margin, and therefore generalization bound, approximating the best possible. One consequence of Corollary 3.2 is that optimizing weakly-regularized logistic loss results in the best possible generalization bound out of all models with the given architecture. This indicates that the widely used algorithm of optimizing deep networks with (cid:96) 2 -regularized logistic loss has an implicit bias towards solutions with good generalization.
Next, we observe that the maximum normalized margin is non-decreasing with the size of the architecture. Formally, for two depth- K architectures M = ( m 1 , . . . , m K − 1 ) and M (cid:48) =
( m (cid:48) 1 , . . . , m (cid:48) K − 1 ) , we say M ≤ M (cid:48) if m i ≤ m (cid:48) i ∀ i = 1 , . . .
K − 1 . Theorem 3.3 states that if M ≤ M (cid:48) , then the max-margin over networks with architecture M (cid:48) is at least the max-margin over networks with architecture M .
Theorem 3.3.
Recall that γ (cid:63), M denotes the maximum normalized margin of a network with architecture M . If M ≤ M (cid:48) , we have γ (cid:63), M ≤ γ (cid:63), M (cid:48) .
As a important consequence, the generalization error bound of Corollary 3.2 for M (cid:48) is at least as good as that for M . This theorem is simple to prove and follows because we can directly implement any network of architecture M using one of architecture M (cid:48) , if M ≤ M (cid:48) .
This can explain why additional overparameterization has been empirically observed to improve generalization in two-layer networks (Neyshabur et al., 2017b): the margin does not decrease with a larger network size, and therefore Corollary 3.2 gives a better generalization bound.
In Section 6, we provide empirical evidence that the test error decreases with larger network size while the margin is non-decreasing. The phenomenon in Theorem 3.3 contrasts with standard (cid:96) 2 -normalized linear prediction. In this setting, adding more features increases the norm of the data, and therefore the generalization error bounds could also increase.
On the other hand, Theorem 3.3 shows that adding more neurons (which can be viewed as learned features) can only improve the generalization of the max-margin solution.  N EURAL NET MAX - MARGIN VS . KERNEL METHODS
We will continue our study of the max-margin neural network via comparison against kernel methods, a context in which margins have already been extensively studied. We show that two-layer networks can obtain a larger margin, and therefore better generalization guarantees, than kernel methods. Our comparison between the two methods is motivated by an equivalence between the (cid:96) 2 max-margin of an inﬁnite-width two-layer network and the (cid:96) 1 -SVM (Zhu et al., 2004) over the lifted feature space deﬁned by the activation function applied to all possible hidden units (Neyshabur et al., 2014; Rosset et al., 2007; Bengio et al., 2006).
The kernel method corresponds to the (cid:96) 2 -SVM in this same feature space, and is equivalent to ﬁxing random hidden layer weights and solving an (cid:96) 2 -SVM over the top layer. In Theorem 4.3, we construct a distribution for which the generalization upper bounds for the (cid:96) 1 -SVM on this feature space are smaller than those for the (cid:96) 2 -SVM by a Ω( √ d ) factor. Our work provides evidence that optimizing all layers of a network can be beneﬁcial for generalization. There have been works that compare (cid:96) 1 and (cid:96) 2 -regularized solutions in the context of feature selection and construct a feature space for which a generalization gap exists (e.g., see Ng (2004)).
In contrast, we work in the ﬁxed feature space of relu activations, which makes our construction particularly challenging.
We will use m to denote the width of the single hidden layer of the network.
Following the convention from Section 3, we will use γ (cid:63),m to denote the maximum possible normalized margin of a two-layer network with hidden layer size m (note the emphasis on the size of the single hidden layer).
The depth K = 2 case of Corollary 3.2 immediately implies that optimizing weakly-regularized (cid:96) 2 loss over width- m two-layer networks gives parameters whose generalization upper bounds depend on the hidden layer size only through 1 /γ (cid:63),m .
Furthermore, from Theorem 3.3 it immediately follows that
[Equation] The work of Neyshabur et al. (2014) links γ (cid:63),m to the (cid:96) 1 SVM over a lifted space. Formally, we deﬁne a lifting function ϕ : R d → L ∞ ( S d − 1 ) mapping data to an inﬁnite feature vector:
[Equation]
where φ is the activation of Section 3.
We look at the margin of linear functionals corresponding to α ∈ L 1 ( S d − 1 ) .
The 1-norm SVM (Zhu et al., 2004) over the lifted feature ϕ ( x ) solves for the maximum margin:
[Equation]
subject to
[Equation]
(4.2) where we rely on the inner product and 1-norm deﬁned in Section 1.2.
This formulation is equivalent to a hard-margin optimization on “convex neural networks” (Bengio et al., 2006).
Bach (2017) also study optimization and generalization of convex neural networks.
Using results from Rosset et al.
; Neyshabur et al.
(2014); Bengio et al. (2006), our Theorem 2.1 implies that optimizing weaklyregularized logistic loss over two-layer networks is equivalent to solving equation 4.2 when the size of the hidden layer is at least n + 1 , where n is the number of training examples. Proposition 4.
essentially restates this with the minor improvement that this equivalence 4 also holds when the size of the hidden layer is n . Proposition 4.1. Let γ (cid:96) 1 be deﬁned in equation 4.2. Then γ (cid:96)
[Equation] = [Equation] For completeness, we prove Proposition 4.1 in Section B, relying on the work of Tibshirani (2013) and Rosset et al.
(2004a). Importantly, the (cid:96) 1 -max margin on the lifted feature space is obtainable by optimizing a ﬁnite neural network. We compare this to the (cid:96) 2 margin attainable via kernel methods.
Following the setup of equation 4.2, we deﬁne the kernel problem over α ∈ L 2 ( S d − 1 ) :
[Equation]
where κ (cid:44) Vol
( S d − 1 ) .
(We scale (cid:107) α (cid:107) 2 by √ κ to make the lemma statement below cleaner.)
First, γ (cid:96) 2 can be used to obtain a standard upper bound on the generalization error of the kernel SVM.
Following the notation of Section 3, we will let L (cid:96) 2 -svm denote the 0-1 population classiﬁcation error for the optimizer of equation 4.3.
Lemma 4.2.
In the setting of Proposition 3.1, with probability at least 1 − δ , the generalization error of the standard kernel SVM with relu feature (deﬁned in equation 4.3) is bounded by
[Equation]
where (cid:15) (cid:96) 2 (cid:44) (cid:115)
[Equation]
is typically a lower-order term.
The bound above follows from standard techniques (Bartlett & Mendelson, 2002), and we provide a full proof in Section C.2.
We construct a data distribution for which this lemma does not give a good bound for kernel methods, but Corollary 3.2 does imply good generalization for two-layer networks.
Theorem 4.3.
There exists a data distribution p data such that the (cid:96) 1 SVM with relu features has a good margin: γ (cid:96) 1 (cid:38) 1 and with probability 1 − δ over the choice of i.i.d.
samples from p data , obtains generalization error
[Equation]
where (cid:15) (cid:96) 1 (cid:44) (cid:113) log(1 /δ )
n is typically a lower order term.
Meanwhile, with high probability the (cid:96) 2 SVM has a small margin: γ (cid:96) 2 (cid:46) max (cid:26)(cid:113) log nn , 1 /d (cid:27) and therefore the generalization upper bound from
Lemma 4.2 is at least
[Equation]
In particular, the (cid:96) 2 bound is larger than the (cid:96) 1 bound by a Ω( √ d ) factor.
Although Theorem 4.3 compares upper bounds, our construction highlights properties of distributions which result in better neural network generalization than kernel method generalization.
Furthermore, in Section 6 we empirically validate the gap in generalization between the two methods.
We brieﬂy overview the construction of p data here.
The full proof is in Section D.1.
Proof sketch for Theorem 4.3.
We base p data on the distribution D of examples ( x, y ) described below.
Here e i is the i-th standard basis vector and we use x (cid:62) e i to represent the i -coordinate of x (since the subscript is reserved to index training examples).
[Equation]
[-1 /y = − 1 , x (cid:62) e 
-]
Figure 1 shows samples from D when there are 3 dimensions.
From the visualization, it is clear that there is no linear separator for D .
As Lemma D.1 shows, a relu network with four neurons can ﬁt this relatively complicated decision boundary.
On the other hand, for kernel methods, we prove that the symmetries in D induce cancellation in feature space.
As a result, the features are less predictive of the true label and the margin will therefore be small.
We formalize this argument in Section D.1.
Gap in regression setting: We are able to prove an even larger Ω( (cid:112) n/d ) gap between neural networks and kernel methods in the regression setting where we wish to interpolate continuous labels.
Analogously to the classiﬁcation setting, optimizing a regularized squared error loss on neural networks is equivalent to solving a minimum 1-norm regression problem (see Theorem D.5).
Furthermore, kernel methods correspond to a minimum 2-norm problem.
We construct distributions p data where the 1-norm solution will have a generalization error bound of O ( (cid:112) d/n ) , whereas the 2norm solution will have a generalization error bound that is Ω(1) and thus vacuous.
In Section D.2, we deﬁne the 1-norm and 2-norm regression problems.
In Theorem D.10 we formalize our construction.
P ERTURBED W ASSERSTEIN GRADIENT FLOW FINDS GLOBAL OPTIMIZERS IN POLYNOMIAL TIME
In the prior section, we studied the limiting behavior of the generalization of a two-layer network as its width goes to inﬁnity.
In this section, we will now study the limiting behavior of the optimization algorithm, gradient descent.
Prior work (Mei et al., 2018; Chizat & Bach, 2018) has shown that as the hidden layer size grows to inﬁnity, gradient descent for a ﬁnite neural network approaches the
Wasserstein gradient ﬂow over distributions of hidden units (deﬁned in equation 5.1).
Chizat & Bach (2018) assume the gradient ﬂow converges, a non-trivial assumption since the space of distributions is inﬁnite-dimensional, and given the assumption prove that Wasserstein gradient ﬂow converges to a global optimizer in this setting, but do not specify a convergence rate. Mei et al. (2018) show global convergence for the inﬁnite-neuron limit of stochastic Langevin dynamics, but also do not provide a convergence rate.
We show that a perturbed version of Wasserstein gradient ﬂow converges in polynomial time .
The informal take-away of this section is that a perturbed version of gradient descent converges in polynomial time on inﬁnite-size neural networks (for the right notion of inﬁnite-size.)
Formally, we optimize the following functional over distributions ρ on R d +1 :
[Equation]
where Φ : R d +1 → R k , R : R k → R , and V : R d +1 → R .
In this work, we consider 2-homogeneous Φ and V .
We will additionally require that R is convex and nonnegative and V is positive on the unit sphere.
Finally, we need standard regularity assumptions on R, Φ , and V :
Assumption 5.1 (Regularity conditions on Φ , R , V ) .
Φ and V are differentiable as well as upper bounded and Lipschitz on the unit sphere.
R is Lipschitz and its Hessian has bounded operator norm.
We provide more details on the speciﬁc parameters (for boundedness, Lipschitzness, etc.)
in Section E.1.
We note that relu networks satisfy every condition but differentiability of Φ .
5 We can ﬁt a neural network under our framework as follows:
Example 5.2 (Logistic loss for neural networks) .
We interpret ρ as a distribution over the parameters of the network.
Let k (cid:44) n and Φ i ( θ ) (cid:44) wφ ( u (cid:62) x i ) for θ =
( w, u ) .
In this case, (cid:82) Φ dρ is a distributional neural network that computes an output for each of the n training examples (like a standard neural network, it also computes a weighted sum over hidden units).
We can compute the distributional version of the regularized logistic loss in equation 3.2 by setting V ( θ ) (cid:44) λ (cid:107) θ (cid:107) 22 and
R ( a 1 , . . . , a n ) (cid:44) (cid:80) ni
=1 log(1 + exp(
− y i a i )) .
We will deﬁne L (cid:48)
[ ρ ] : R d +1 → R with L (cid:48)
[ ρ ]( θ ) (cid:44) (cid:104) R (cid:48)
( (cid:82) Φ dρ ) , Φ( θ ) (cid:105)
+ V ( θ ) and v [ ρ ]( θ )
(cid:44) −∇ θ L (cid:48)
[ ρ ]( θ ) .
Informally, L (cid:48)
[ ρ ] is the gradient of L with respect to ρ , and v is the induced velocity ﬁeld.
For the standard Wasserstein gradient ﬂow dynamics, ρ t evolves according to
[Equation]
where ∇· denotes the divergence of a vector ﬁeld.
For neural networks, these dynamics formally deﬁne continuous-time gradient descent when the hidden layer has inﬁnite size (see Theorem 2. of Chizat & Bach (2018), for instance).
We propose the following modiﬁcation of the Wasserstein gradient ﬂow dynamics:
[Equation]
where U d is the uniform distribution on S d .
In our perturbed dynamics, we add very small uniform noise over U d , which ensures that at all time-steps, there is sufﬁcient mass in a descent direction for the algorithm to decrease the objective.
For inﬁnite-size neural networks, one can informally interpret this as re-initializing a very small fraction of the neurons at every step of gradient descent.
We prove convergence to a global optimizer in time polynomial in 1 /(cid:15), d , and the regularity parameters.
Theorem 5.3 (Theorem E.4 with regularity parameters omitted) .
Suppose that Φ and V are 2homogeneous and the regularity conditions of Assumption 5.1 are satisﬁed.
Also assume that from starting distribution ρ 0 , a solution to the dynamics in equation 5.2 exists.
Deﬁne L (cid:63) (cid:44) inf ρ L [ ρ ] .
Let (cid:15) > 0 be a desired error threshold and choose σ (cid:44) exp( − d log(1 /(cid:15) )
poly ( k, L [ ρ 0 ] − L (cid:63) )) and t (cid:15) (cid:44) d 2 (cid:15) 4 poly (log(1 /(cid:15) ) , k, L [ ρ 0 ] − L (cid:63) ) , where the regularity parameters for Φ , V , and R are hidden in the poly ( · ) .
Then, perturbed Wasserstein gradient ﬂow converges to an (cid:15) -approximate global minimum in t (cid:15) time:
[Equation]
We provide a theorem statement that includes regularity parameters in Section E.1.
We prove the theorem in Section E.2.
As a technical detail, Theorem 5.3 requires that a solution to the dynamics exists.
We can remove this assumption by analyzing a discrete-time version of equation 5.2:
[Equation]
and additionally assuming Φ and V have Lipschitz gradients.
In this setting, a polynomial time convergence result also holds.
We state the result in Section E.3.
An implication of our Theorem 5.3 is that for inﬁnite networks, we can optimize the weaklyregularized logistic loss in time polynomial in the problem parameters and λ − 1 .
By Theorem 2.2, we only require λ − 1 = poly ( n ) to approximate the maximum margin within a constant factor.
Thus, for inﬁnite networks, we can approximate the max margin within a constant factor in polynomial time.
S IMULATIONS
We ﬁrst compare the generalization of neural networks and kernel methods for classiﬁcation and regression.
In Figure 2 we plot the generalization error and predicted generalization upper bounds 6 of a trained neural network against a (cid:96) 2 kernel method with relu features as we vary n .
Our data comes from a synthetic distribution generated by a neural network with 6 hidden units; we provide a detailed setup in Section F.1.
For classiﬁcation we plot 0-1 error, whereas for regression we plot squared error.
The variance in the neural network generalization bound for classiﬁcation likely occured because we did not tune learning rate and training time, so the optimization failed to ﬁnd the best margin.
The plots show that two-layer networks clearly outperform kernel methods in test error as n grows.
However, there seems to be looseness in the bounds: the kernel generalization bound appears to stay constant with n (as predicted by our theory for regression), but the test error decreases.
We also plot the dependence of the test error and margin on the hidden layer size in Figure 3 for synthetic data generated from a ground truth network with 10 hidden units and also MNIST.
The plots indicate that test error is decreasing in hidden layer size while margin is increasing, as Theorem 3. predicts. We provide more details on the experimental setup in Section F.2. In Section F.3, we verify the convergence of a simple neural network to the max-margin solution as regularization decreases.
In Section F.4, we train modiﬁed WideResNet architectures on CIFAR and CIFAR100.
Although ResNet is not homogeneous, we still report improvements in generalization from annealing the weight decay during training, versus staying at a ﬁxed decay rate.
C ONCLUSION
We have made the case that maximizing margin is one of the inductive biases of relu networks obtained from optimizing weakly-regularized cross-entropy loss.
Our framework allows us to directly analyze generalization properties of the network without considering the optimization algorithm used to obtain it. Using this perspective, we provide a simple explanation for why over-parametrization can improve generalization.
It is a fascinating question for future work to characterize other generalization properties of the max-margin solution.
On the optimization side, we make progress towards understanding over-parametrized gradient descent by analyzing inﬁnite-size neural networks.
A natural direction for future work is to apply our theory to optimize the margin of ﬁnite-sized neural networks.
A M ISSING P ROOFS IN S ECTION
We ﬁrst show that L λ does indeed have a global minimizer.
Claim A.1.
In the setting of Theorems 2.1 and A.3, arg min Θ L λ (Θ) exists.
Proof.
We will argue in the setting of Theorem 1 where L λ is the multi-class cross entropy loss, because the logistic loss case is analogous.
We ﬁrst note that L λ is continuous in Θ because f is continuous in Θ and the term inside the logarithm is always positive.
Next, deﬁne b (cid:44) inf Θ L λ (Θ) > 0 .
Then we note that for (cid:107) Θ (cid:107) >
( b/λ ) 1 /r (cid:44) M , we must have L λ (Θ) > b .
It follows that inf (cid:107) Θ (cid:107)≤ M L λ (Θ) =
inf Θ L λ (Θ) .
However, there must be a value Θ λ which attains inf (cid:107) Θ (cid:107)≤ M L λ (Θ) , because { Θ : (cid:107) Θ (cid:107) ≤ M } is a compact set and L λ is continuous.
Thus, inf Θ L λ (Θ) is attained by some Θ λ .
A. M ISSING P ROOFS FOR M ULTI - CLASS S ETTING
Towards proving Theorem 2.1, we ﬁrst show as we decrease λ , the norm of the solution (cid:107) Θ λ (cid:107) grows.
Lemma A.2.
In the setting of Theorem 2.1, as λ → 0 , we have (cid:107) Θ λ (cid:107) → ∞ .
To prove Theorem 2.1, we rely on the exponential scaling of the cross entropy: L λ can be lower bounded roughly by exp( −(cid:107) Θ λ (cid:107) γ λ ) , but also has an upper bound that scales with exp( −(cid:107) Θ λ (cid:107) γ (cid:63) ) .
By Lemma A.2, we can take large (cid:107) Θ λ (cid:107) so the gap γ (cid:63) − γ λ vanishes.
This proof technique is inspired by that of Rosset et al.
(2004a).
Proof of Theorem 2.1.
For any M > 0 and Θ with γ Θ (cid:44) min i (cid:0) f
[Equation]
¯Θ; x i [Equation] − max j (cid:54) = y i f
[Equation]
¯Θ; x i ) [Equation] [Equation] (by the homogeneity of f )
[Equation]
We can also apply (cid:80) j (cid:54) =
y i exp( M a ( f j (Θ; x i )
− f y i (Θ; x i )))
≥ max exp( M a ( f j (Θ [Equation] x i ) − f y
i (Θ; x i ))) = exp γ Θ in order to lower bound equation A.1 and obtain
[Equation]
Applying equation A.2 with M = (cid:107) Θ λ (cid:107) and Θ = Θ (cid:63) , noting that (cid:107) Θ (cid:63) (cid:107) ≤ 1 , we have:
[Equation]
Next we lower bound L λ (Θ λ ) by applying equation A.3,
[Equation]
Combining equation A.4 and equation A.5 with the fact that L λ (Θ λ ) ≤ L λ (Θ (cid:63) (cid:107) Θ λ (cid:107) )
(by the global optimality of Θ λ ), we have
[Equation]
Recall that by Lemma A.2, as λ → 0 , we have (cid:107) Θ λ (cid:107) → ∞ .
Therefore, exp( −(cid:107) Θ λ (cid:107) a γ (cid:63) ) , exp( −(cid:107) Θ λ (cid:107) a γ λ ) → 0 .
Thus, we can apply Taylor expansion to the equation above with respect to exp( −(cid:107) Θ λ (cid:107) a γ (cid:63) ) and exp( −(cid:107) Θ λ (cid:107) a γ λ ) .
If max { exp( −(cid:107) Θ λ (cid:107) a γ (cid:63) ) , exp( −(cid:107) Θ λ (cid:107) a γ λ ) } < 1 , then we obtain n ( l − 1) exp( −(cid:107) Θ λ (cid:107) a γ [Equation] ) ≥ [Equation]
Θ λ
[Equation]
a γ λ ) − O
[Equation]
max
[Equation]
−(cid:107)
Θ λ
[Equation] [Equation] a γ λ ) 2 } ) We claim this implies that γ (cid:63) ≤ lim inf λ → 0 γ λ .
If not, we have lim inf λ → 0 γ λ < γ (cid:63) , which implies that the equation above is violated with sufﬁciently large (cid:107) Θ λ (cid:107) ( (cid:107) Θ λ (cid:107) (cid:29) log(
( (cid:96) − 1) n ) 1 /a would sufﬁce).
By Lemma A.2, (cid:107) Θ λ (cid:107) → ∞ as λ → 0 and therefore we get a contradiction.
Finally, we have γ λ ≤ γ (cid:63) by deﬁnition of γ (cid:63) .
Hence, lim λ → 0 γ λ exists and equals γ (cid:63) .
Now we ﬁll in the proof of Lemma A.2.
Proof of Lemma A.2.
For the sake of contradiction, we assume that ∃ C > 0 such that for any λ 0 > 0 , there exists 0 < λ < λ 0 with (cid:107) Θ λ (cid:107) ≤ C .
We will determine the choice of λ 0 later and pick λ such that (cid:107) Θ λ (cid:107) ≤ C .
Then the logits (the prediction f j (Θ , x i ) before softmax) are bounded in absolute value by some constant (that depends on C ), and therefore the loss function − log
[Equation]
for every example is bounded from below by some constant D > 0 (depending on C but not λ .)
Let M = λ − 1 / ( r +1) , we have that
[Equation]
(by the optimality of Θ λ )
[Equation]
(by equation A.2)
[Equation]
Taking a sufﬁciently small λ 0 , we obtain a contradiction and complete the proof.
A. F ULL B INARY C LASSIFICATION S ETTING
For completeness, we state and prove our max-margin results for the setting where we ﬁt binary labels y i ∈ {− 1 , +1 } (as opposed to indices in [ l ] ) and redeﬁning f (Θ; · ) to assign a single real-valued score (as opposed to a score for each label).
This lets us work with the simpler λ -regularized logistic loss:
[Equation]
As before, let Θ λ ∈ arg min L λ (Θ) , and deﬁne the normalized margin γ λ by γ λ (cid:44) min
i y i f (¯Θ λ ; x i ) .
Deﬁne the maximum possible normalized margin
[Equation]
Theorem A.3.
Assume γ (cid:63) > 0 in the binary classiﬁcation setting with logistic loss.
Then as λ → 0 , γ λ → γ (cid:63) .
The proof follows via simple reduction to the multi-class case.
Proof of Theorem A.3.
We prove this theorem via reduction to the multi-class case with l = 2 .
Construct ˜ f
: R d → R 2 with ˜ f 1 (Θ; x i ) = − 12 f
(Θ; x i ) and ˜ f 2 (Θ; x i ) =
12 f (Θ; x i ) .
Deﬁne new labels ˜ y
i =
if y i = − 1 and ˜ y i
=
if y i = 1 .
Now note that ˜ f
˜ y
i (Θ; x i ) − ˜ f
j (cid:54) =
˜
y
i (Θ; x i ) =
y i f
(Θ; x i ) , so the multi-class margin for Θ under ˜ f is the same as binary margin for Θ under f .
Furthermore, deﬁning
[Equation]
we get that ˜ L λ (Θ) =
L λ (Θ) , and in particular, ˜ L λ and L λ have the same set of minimizers.
Therefore we can apply Theorem 2.1 for the multi-class setting and conclude γ λ → γ (cid:63) in the binary classiﬁcation setting.
A. M ISSING P ROOF FOR O PTIMIZATION A CCURACY
Proof of Theorem 2.2.
Choose B (cid:44) (cid:16) 1 γ (cid:63) log ( l − 1)( γ (cid:63) )
r/a λ (cid:17) 1 /a .
We can upper bound L λ (Θ (cid:48) ) by computing
[Equation]
(by equation A.2) ≤ β ( l − 1) exp( − B a γ (cid:63) )
+ βλB r (using log(1 + x ) ≤ x ) ≤ β λ ( γ (cid:63) ) r/a + βλ (cid:18)γ (cid:63) log ( l − 1) ( γ (cid:63) ) r/a λ (cid:19) r/a ≤ β λ ( γ (cid:63) ) r/a (cid:32) 1 + (cid:18) log ( l − 1) ( γ (cid:63) ) r/a λ (cid:19) r/a (cid:33) (cid:44)
L ( UB )
Furthermore, it holds that (cid:107) Θ (cid:48) (cid:107) r ≤ L ( UB ) λ .
Now we note that
[Equation]
for sufﬁciently large c depending only on a/r .
Now using the fact that log( x ) ≥ x 1+ x ∀ x ≥ − 1 , we additionally have the lower bound L λ (Θ
[Equation]
Since L ( UB ) ≤ 1 , we can rearrange to get
[Equation]
The middle inequality followed because x 1 − x is increasing in x for 0 ≤ x < 1 , and the last because
L ( UB ) ≤ 12 n .
Since − log 2 nL ( UB ) > 0 we can also apply the bound (cid:107) Θ (cid:48) (cid:107) r ≤ L
[Equation]
UB
[Equation]
λ to get
[Equation]
UB )
[Equation]
  We will ﬁrst bound ♣ . First note that
[Equation]
) where the last inequality follows from the fact that ( γ (cid:63) ) r/a λ ≥ n c ( l − 1) c and β ≤ 2 . Next, using the fact that log ( γ (cid:63) ) r/a λ ≥ 1 (2 r/a
[Equation]
1) a/r , we note that
[Equation]
A
[Equation]
Combining equation A.7 and equation A.8, we can conclude that
[Equation]
Finally, we note that if 1 + (cid:16) log ( l − 1)( γ (cid:63) ) r/a λ (cid:17) r/a is a sufﬁciently large constant that depends only on a/r (which can be achieved by choosing c sufﬁciently large), it will follow that ♥ ≤ 1 10 . Thus, if c ≥ 5 , we can combine our bounds on ♣ and ♥ to get that
[Equation]
Proposition 4.1 follows simply from applying Corollary 1 of Neyshabur et al.
(2014) to a hard-margin
SVM problem.
For completeness, we provide another proof here.
The proof of Proposition 4.1 will consist of two steps: ﬁrst, show that equation 4.2 has an optimal solution with sparsity n , and second, show that sparse solutions to equation 4.2 can be mapped to a neural network with the same margin, and vice versa.
The following lemma and proof are based on Lemma 14 of Tibshirani (2013).
Lemma B.1.
Let supp ( α ) (cid:44) { ¯ u : | α (¯ u ) | > 0 } .
There exists an optimal solution α (cid:63) to equation 4.
with | supp ( α (cid:63) ) | ≤ n .
For the proof of this lemma, we ﬁnd it convenient to work with a minimum norm formulation which we show is equivalent to equation 4.2: [Equation]
subject to y i (cid:104)
[Equation]
i (B.1)
Claim B.2.
Let S ⊂ L 1 ( S d − 1 ) be the set of optimizers for equation 4 . 2 , and let S (cid:48)
⊂
L 1 ( S d − 1 ) be the set of optimizers for equation B.1.
If equation B.1 is feasible, for any α ∈ S , αγ (cid:96) 1 ∈ S (cid:48) , and for any α (cid:48) ∈ S (cid:48) , α (cid:48) (cid:107) α (cid:48)
(cid:107) 1 ∈ S .
Proof.
Let opt (cid:48) denote the optimal objective for equation B.1.
We note that
[Equation]
(cid:107) 1 is feasible for equation 4.2 with objective 1opt (cid:48) , and therefore γ (cid:96)≥ 1opt (cid:48) .
Furthermore, 1 2 γ (cid:96) 1 y i (cid:82) ¯ u ∈ S d − 1 α (¯ u ) φ (¯ u (cid:62) x i ) d ¯ u ≥ 1 ∀ i , and so αγ (cid:96) 1 is feasible for equation B.1 with objective 1 γ (cid:96) 1 .
Therefore, opt (cid:48) ≤ 1 γ (cid:96) 1 .
As a result, it must hold that opt (cid:48)
= 1 γ (cid:96) 1 , which means that α (cid:48) (cid:107) α (cid:48) (cid:107) 1 is optimal for equation 4.2, and αγ (cid:96) 1 is optimal for equation B.1, as desired.
First, note that if equation B.1 is not feasible, then γ (cid:96) 1 = 0 and equation 4.2 has a trivial sparse solution, the all zeros function.
Thus, it sufﬁces to show that an optimal solution to equation B. exists that is n -sparse, since by Lemma B.2 equation B.1 and equation 4.2 have equivalent solutions up to a scaling.
We begin by taking the dual of equation B.1.
Claim B.3.
The dual of equation B.1 has form
[Equation]
subject to (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
For any primal optimal solution α (cid:63) and dual optimal solution λ (cid:63) , it must hold that n (cid:88) i =
[Equation]
Proof.
The dual form can be solved for by computation.
By strong duality, equation B.2 must follow from the KKT conditions.
Now deﬁne the mapping v : S d − 1 → R n with v i (¯ u ) (cid:44)
y i φ (¯ u (cid:62) x i ) .
We will show a general result about linearly dependent v (¯ u ) for ¯ u ∈ supp ( α (cid:63) ) , after which we can reduce directly to the proof of
Tibshirani (2013).
Claim B.4.
Let α (cid:63) be any optimal solution.
Suppose that there exists S ⊆ supp ( α (cid:63) ) such that { v (¯ u ) : ¯ u ∈ S } forms a linearly dependent set, i.e.
[Equation]
for coefﬁcients c .
Then (cid:80) ¯
u ∈ S
c ¯ u sign( α (cid:63) (¯ u )) = 0 .
Proof.
Let λ (cid:63) be any dual optimal solution, then λ (cid:63) (cid:62) v (¯ u ) =
sign( α (cid:63) (¯ u )) ∀ ¯ u ∈ supp
( α (cid:63) ) by Claim
B.3.
Thus, we apply λ (cid:63) (cid:62) to both sides of equation B.3 to get the desired statement.
Proof of Lemma B.1.
The rest of the proof follows Lemma 14 in Tibshirani (2013).
The lemma argues that if the conclusion of Claim B.4 holds and an optimal solution α (cid:63) has S ⊆ supp ( α (cid:63) ) with { v (¯ u ) : ¯ u ∈ S } linearly dependent, we can construct a new α (cid:48) with (cid:107) α (cid:48) (cid:107)
= (cid:107) α (cid:63) (cid:107) 1 and supp ( α (cid:48) )
⊂ supp ( α (cid:63) )
(where the inclusion is strict).
Thus, if we consider an optimal α (cid:63) with minimal support, it must follow that { v (¯ u ) : ¯ u ∈ supp ( α (cid:63) ) } is a linearly independent set, and
therefore | supp ( α (cid:63) ) | ≤ n .
We can now complete the proof of Proposition 4.1.
Proof of Proposition 4.1.
For ease of notation, we will parametrize a two-layer network with m units by top layer weights w 1 , . . . , w m ∈ R and bottom layer weights u 1 , . . . , u m ∈ R d .
As before, we use Θ to refer to the collection of parameters, so the network computes the real-valued function
[Equation] Note that we simply renamed the variables from the parametrization of equation 3.1.
We ﬁrst apply Lemma B.1 to conclude that equation 4.2 admits a n -sparse optimal solution α (cid:63) . Because of sparsity, we can now abuse notation and treat α (cid:63) as a real-valued function such that [Equation]
|
[Equation]
¯ u ) | ≤ 1 .
We construct Θ corresponding to a two-layer network with m ≥ n hidden units and normalized margin at least γ (cid:96) 1 2 . For clarity, we let W correspond to the top layer weights and U correspond to the bottom layer weights.
For every ¯ u ∈ supp ( α ) , we let Θ have a corresponding hidden unit j with ( w j , u j ) =
[Equation]
sign
[Equation]
, and set the remaining hidden units to (cid:126) 0 .
This is possible because m ≥ n .
Now
[Equation]
Furthermore,
[Equation]
Thus it follows that Θ has normalized margin at least γ (cid:96) 1 / 2 , so γ (cid:63),m ≥ γ (cid:96) 1 / 2 .
To conclude, we show that γ (cid:63),m ≤ γ (cid:96) 1 / 2 .
Let Θ (cid:63),m denote the parameters obtaining optimal m -unit margin γ (cid:63),m with hidden units ( w (cid:63),mj , u (cid:63),mj ) for j ∈ [ m ] .
We can construct α to put a scaled delta mass of 2 w (cid:63),mj (cid:107) u (cid:63),mj (cid:107) 2 on ¯ u (cid:63),mj for j ∈ [ m ] .
It follows that {+(cid:107) α (cid:107) 
+}
= m (cid:88)
j = 2 {+| w (cid:63),mj |(cid:107) u (cid:63),mj (cid:107) 2 ≤ m (cid:88) j = w (cid:63),mj 2 + (cid:107) u (cid:63),mj (cid:107) 
 = (cid:107) Θ+} (cid:63),m (cid:107) 22 ≤
Furthermore, (cid:90) S d − 1 α (¯ u ) φ (¯ u (cid:62) x ) =m (cid:88) j = w (cid:63),mj (cid:107) u (cid:63),mj (cid:107) 2 φ ((¯ u (cid:63),mj ) (cid:62) x )
=m (cid:88) j = w (cid:63),mj φ ( u (cid:63),mj (cid:62) x )
= 2 f (Θ (cid:63),m ; x ) Thus, α is a feasible solution to equation 4.2 with objective value at least 2 γ (cid:63),m . Therefore, γ (cid:96) 1 ≥ 2 γ (cid:63),m , so γ (cid:63),m = γ (cid:96) 1 / 2 .
C R ADEMACHER C OMPLEXITY AND G ENERALIZATION E RROR
We prove the generalization error bounds stated in Proposition 3.1 and Lemma 4.2 via Rademacher complexity and margin theory.
Assume that our data X, Y are drawn i.i.d.
from ground truth distribution p data supported on X × Y .
For some hypothesis class F of real-valued functions, we deﬁne the empirical Rademacher complexity ˆ
R ( F ) as follows:
[Equation]
where (cid:15) i are independent Rademacher random variables.
For a classiﬁer f , following the notation of Section 3 we will use L ( f ) (cid:44) Pr ( x,y ) ∼ p data ( yf ( x ) ≤ 0) to denote the population 0-1 loss of the classiﬁer f .
The following classical theorem (Koltchinskii et al., 2002), (Kakade et al., 2009) bounds generalization error in terms of the Rademacher complexity and margin loss.
Theorem C.1 (Theorem 2 of Kakade et al.
Let ( x i , y i ) ni
=1 be drawn iid from p data .
We work in the binary classiﬁcation setting, so Y
= {− 1 , 1 } .
Assume that for all f ∈ F , we have sup x ∈X f ( x ) ≤ C . Then with probability at least 1 − δ over the random draws of the data, for every γ > 0 and f ∈ F , [Equation] C. P ROOF OF P ROPOSITION 3.
We will prove Proposition 3.1 by applying the Rademacher complexity bounds of Golowich et al.
(2017) with Theorem C.1.
First, we show the following lemma bounding the generalization of neural networks whose weight matrices have bounded Frobenius norms. Lemma C.2. Deﬁne the hypothesis class F K over depth- K neural networks by
[Equation]
Let C (cid:44) sup x ∈X (cid:107) x (cid:107) 2 . Recall that L (Θ) denotes the 0-1 population loss L ( f (Θ; · )) . Then for any f (Θ; · ) ∈ F K classifying the training data correctly with unnormalized margin γ Θ (cid:44) min i y i f (Θ; x i ) > 0 , with probability at least 1 − δ ,
[Equation]
Note the dependence on the unnormalized margin rather than the normalized margin.
Proof. We ﬁrst claim that sup f (Θ; · ) ∈F K sup x ∈X f (Θ; x ) ≤ C . To see this, for any f (Θ; · ) ∈ F K , f (Θ; x ) = W K φ ( · · · φ ( W 1 x ) · · · )
[Equation]
K
[Equation]
F (cid:107)
[Equation]
( W K − 1 φ ( · · · φ ( W 1 x ) · · · ) (cid:107)≤ (cid:107) W K (cid:107) F (cid:107) W K − 1 φ ( · · · φ ( W 1 x ) · · · ) (cid:107)(since φ is 1-Lipschitz and φ (0) = 0 , so φ performs a contraction)
[Equation]
2 ≤ C (repeatedly applying this argument and using (cid:107) W j (cid:107) F < 1 ) Furthermore, by Theorem 1 of Golowich et al. (2017), ˆ R ( F K ) has upper bound
[Equation]
Thus, we can apply Theorem C.1 to conclude that for all f (Θ; · ) ∈ F K and all γ > 0 , with probability 1 − δ ,
[Equation]
In particular, by deﬁnition choosing γ = γ Θ makes the ﬁrst term on the LHS vanish and gives the statement of the lemma. Proof of Proposition 3.1. Given parameters Θ =
( W 1 , . . .
, W K ) , we ﬁrst construct parameters ˜Θ =
( ˜ W 1 , . . .
, ˜ W K ) such that f (¯Θ; · ) and f (˜Θ; · ) compute the same function, and (cid:107) ˜ W 1 (cid:107) 2 F = (cid:107) ˜
W 2 (cid:107) 2 F = · · · = (cid:107) ˜ W K (cid:107) 2 F ≤ 1 K .
To do this, we set ˜
[Equation]
By construction
[Equation]
the AM-GM inequality)
Furthermore, we also have
[Equation]
(by the homogeneity of φ )
[Equation]
(since f is K -homogeneous in Θ )
[Equation]
Now we note that by construction, L (Θ)
=
L ( ˜ Θ) .
Now f ( ˜ Θ; · ) must also classify the training data perfectly, has unnormalized margin γ , and furthermore f (˜Θ; · ) ∈ F K . As a result, Lemma C.2 allows us to conclude the desired statement. To conclude Corollary 3.2, we apply the above on Θ λ, M and use Theorem A.3. C. P ROOF OF K ERNEL G ENERALIZATION B OUNDS
Let F 2 ,φB denote the class of (cid:96) 2 -bounded linear functionals in lifted feature space: F 2 ,φB (cid:44) { x (cid:55)→ (cid:104) α, ϕ ( x ) (cid:105) :
α ∈ L 2 ( S d − 1 ) , (cid:107) α (cid:107) 2 ≤ B } .
We abuse notation and write α ∈ F 2 ,φ B to indicate a linear functional from F 2 ,φB .
As before, we will use L ( α ) to indicate the 0-1 population loss of the classiﬁer x (cid:55)→ (cid:104) α, ϕ ( x ) (cid:105) and let C (cid:44) sup x ∈X (cid:107) x (cid:107) 2 be an upper bound on the norm of the data.
We focus on analyzing the Rademacher complexity ˆ R ( F 2 ,φB ) , mirroring derivations done in the past (Bartlett & Mendelson, 2002).
We include our derivations here for completeness.
Lemma
[Equation] [Equation]
Proof.
We write
[Equation]
= (cid:15) i (cid:15) j (cid:104)
[Table]
( x i ) ,
[Table]
( x i ) (cid:105) 

[Equation]
(cid:118)(cid:117)(cid:117)(cid:116)
n
[Table]
i = (cid:107)
[Table]
[Table]
i ) (cid:107)(terms where i (cid:54) =
j cancel out)
As an example, we can apply this bound to relu features:
Corollary C.4.
Suppose that φ is the relu activation.
Let κ (cid:44) Vol ( S d − 1 ) .
Then ˆ R ( F 2 ,φB ) (cid:46) [Equation] X [Equation]
Proof.
We ﬁrst show that (cid:107) ϕ
( x i ) (cid:107) 22 =
Θ (cid:0) κd (cid:107) x
i (cid:107) 22 (cid:1) .
We can compute (cid:107)[-ϕ ( x i ) (cid:107) 
 = Vol ( S d − 1 )-]
[Equation]
( M 2 is the second moment of N (0 , 1) )
[Equation]
C.2) where the last line uses the computation provided in Lemma A.1 by Du et al.
Now we plug this into Lemma C.3 to get the desired bound.
We will now prove Lemma 4.2.
Proof of Lemma 4.2.
From equation C.2, we ﬁrst obtain sup x ∈X (cid:107) ϕ
( x ) (cid:107) 2 (cid:46) C (cid:112) κd .
Denote the optimizer for equation 4.3 by α (cid:96) 2 .
Note that √ κα (cid:96) 2 ∈ F 2 ,φ 1 , and furthermore L ( α (cid:96) 2 ) =
L ( √ κα (cid:96) 2 ) .
Since √ κα (cid:96) 2 has unnormalized margin √ κγ (cid:96) 2 , we apply Theorem C.1 on margin √ κγ (cid:96) 2 and hypothesis class F 2 ,φto get with probability 1 − δ ,
[Equation]
(applying Corollary C.4)
D M ISSING P ROOFS FOR C OMPARISON TO K ERNEL M ETHODS D. C LASSIFICATION
In this section we will complete a proof of Theorem 4.3.
Recall the construction of the distribution D provided in Section 4.
We ﬁrst provide a classiﬁer of this data with small (cid:96) 1 norm.
Lemma D.1.
In the setting of Theorem 4.3, we have that
[Equation]
Proof.
Consider the network f ( x ) = 14 (cid:16) ( x (cid:62) ( e 1 + e 2 ) / √ 2) +
+( x (cid:62) ( − e 1 − e 2 ) / √ 2) + − ( x (cid:62) ( − e 1 + e 2 ) / √ 2) + − ( x (cid:62) ( e 1 − e 2 ) / √ 2) + (cid:17) .
The attained margin γ = √ 24 , so γ (cid:96) 1 ≥ √ 24 .
Now we will upper bound the margin attainable by the (cid:96) 2 SVM. Lemma D.2 (Margin upper bound tool) . In the setting of Theorem 4.3, we have
[Equation]
Proof.
By the deﬁnition of γ (cid:96) 2 , we have that for any α with √ κ (cid:107) α (cid:107) 2 ≤ 1 , we have [Equation] n n (cid:88) i = (cid:104)
[Equation]
y
[Equation]
ϕ ( x [Equation] ) (cid:105) Setting α = 1 √ κ 1 n (cid:80) ni =1 ϕ ( x i )
y i / (cid:107) 1 n (cid:80) ni
=1 ϕ ( x i )
y i (cid:107) 2 completes the proof.
(Attentive readers may realize that this is equivalent to setting the dual variable of the convex program 4.3 to all 1’s function.) Lemma D.3.
In the setting of Theorem 4.3, let ( x i , y i ) ni
=1 be n i.i.d samples and corresponding labels from D .
Let ϕ be deﬁned in equation 4.1 with φ = relu . With high probability (at least 1 − dn − 10 ), we have [Equation] Proof.
Let W i = ϕ
( x i )
y i .
We will bound several quantities regarding W i ’s.
In the rest of the proof, we will condition on the event E that ∀ i, (cid:107) x i (cid:107)
(cid:46) d log n .
Note that E is a high probability event and conditioned on E , x i ’s are still independent.
We omit the condition on E in the rest of the proof for simplicity.
We ﬁrst show that assuming the following three inequalities that the conclusion of the Lemma follows.
∀
i, (cid:107) W
i (cid:107) 22 (cid:46) κ log n .
σ 2 (cid:44) Var[ (cid:80)
i W i ] (cid:44) (cid:80) ni
=1 E [ (cid:107) W
i −
E W i (cid:107) 22 ] (cid:46) nκ log n 3.
(cid:107) E [ (cid:80) W
i ] (cid:107)
(cid:46) √ κn/d .
By bullets 1, 2, and Bernstein inequality, we have that with probability at least 1 − dn − 10 over the randomness of the data ( X, Y ) , [Equation] n [Equation]
i
[Equation]
By bullet 3 and equation above, we complete the proof with triangle inequality:
[Equation]
Therefore, it sufﬁces to prove bullets 1, 2 and 3.
Note that 2 is a direct corollary of 1 so we will only prove 1 and 3.
We start with 3:
By the deﬁnition of the (cid:96) 2 norm in L 2 ( S d − 1 ) and the independence of ( x i , y i ) ’s, we can rewrite
[Equation]
Let ¯ u
=
(¯ u 1 , . .
, ¯ u d ) and ¯ u − 2 = (¯ u 3 , . .
, ¯ u d ) ∈ R d − 2 , and deﬁne τ (cid:44) (cid:107) ¯ u − 2 (cid:107) 2 .
Let x − 2 =
( x (cid:62) e 3 , . . .
, x (cid:62) e d ) .
Note that ϕ
( x )[¯ u ] y =
y [¯ u 1 · x (cid:62) e
+ ¯ u 2 · x (cid:62) e
+ ¯ u (cid:62)− 2 x − 2 ] and ¯ u (cid:62)− 2 x − 2 has distribution (cid:107) ¯ u − 2 (cid:107) 2 · N (0 , 1) =
τ · N (0 , 1) .
Let z = ¯ u (cid:62)− 2 x − 2 /τ , and therefore z has standard normal distribution.
With this change of the variables, by the deﬁnition of the distribution D , we have
[Equation]
By claim D.4, and the 1-homogeneity of relu, we can simplify the above equation to
[Equation]
It follows that
[Equation]
Combining equation D
[Equation] 1 and equation D [Equation] 2 we complete the proof of bullet 3.
Next we prove bullet 1.
Note that ϕ
( x )[¯ u ] y is bounded by | ¯ u 1 | + | ¯ u 2 | + (cid:107) ¯ u (cid:62)− 2 x − 2 (cid:107) 2 .
Therefore, conditioned on
[Equation]
x i (cid:107)
[Equation]
d
[Equation]
Hence we complete the proof.
Claim D.4.
Let Z ∼ N (0 , 1) and a ∈ R .
Then, there exists a universal constant c 1 and c 2 such that | E [( a + Z )
+ + ( − a + Z ) + ] − 2 c 1 | ≤ c 2 min {| a | , a 2 } .
Proof.
Without loss of generality we can assume a ≥ 0 .
Then,
E [( a + Z ) + + ( − a + Z ) + ] =
E [( a + Z {+) 
+}
[ Z ≥ − a ]] + E
[( Z − a ) 1 [ Z ≥ a ]]
=
E [ a · 1 [ Z ≥ − a ]] + E [ Z · 1 [ Z ≥ − a ]] − E [ a · 1 [ Z ≥ a ]] + E [ Z · 1 [ Z ≥ a ]] =
E [ a 1 [ − a ≤ Z ≤ a ]] + 2 E [ Z · 1 [ Z ≥ a ]] (by E [ Z · 1 [ − a ≤ Z ≤ a ]] = 0 )
=
E [ a 1 [ − a ≤ Z ≤ a ]] + 2 E
[ Z · 1 [ Z ≥ 0]]
− 2 E
[ Z · 1 [ a ≥ Z ≥ 0]]
= 2 c 1 + O (min { a, a 2 } ) where the last equality uses the fact that c 1 (cid:44) E [ Z · 1 [ Z ≥ 0]] and E [ a 1 [ − a ≤ Z ≤ a ]] ≤ a E [ 1 [ − a ≤ Z ≤ a ]] (cid:46) a min { 1 , a } .
Now we will prove Theorem 4.3.
Proof of Theorem 4.3.
To circumvent the technical issue of bounded support in Proposition 3.1 and
Lemma 4.2, we construct p data to be a slightly modiﬁed version of D : perform rejection sampling of ( x, y ) ∼ D until we obtain a sample with (cid:107) x (cid:107) {+
+} (cid:46) d log n .
Since this occurs with very high probability, the high probability result of Lemma D.3 still translates to p data .
Now apply Lemma D. to conclude that γ (cid:96) 2 (cid:46) log n √ n + 1 d .
Furthermore, Lemma D.1 allows us to conclude that γ (cid:96) 1 (cid:38) 1 .
We can therefore apply Proposition 3.1, and conclude that with probability 1 − δ ,
[Equation]
Furthermore, plugging γ (cid:96) 2 into the bound of Lemma 4.2 gives us
[Equation]
D. R EGRESSION
We will ﬁrst deﬁne the 1-norm and 2-norm regression problems.
The regression equivalent of equation 4.2 for α ∈ L 1 ( S d − 1 ) is as follows:
[Equation]
subject to (cid:104) [Equation] , ϕ (
[Equation]
Next we deﬁne the regression version of equation 4.3:
[Equation]
subject to (cid:104)
[Equation]
ϕ [Equation]
D [Equation] where α ∈ L 2 ( S d − 1 ) .
We will brieﬂy motivate our study of the regression setting by connecting the minimum 1-norm solution to neural networks.
To compare, in the classiﬁcation setting, optimizing the weakly regularized loss over neural networks is equivalent to solving the (cid:96) 1 SVM.
In the regression setting, solving the weakly regularized squared error loss is equivalent is equivalent to ﬁnding the minimum 1-norm solution that ﬁts the datapoints exactly.
Theorem D.5.
Let f (Θ; · ) be some two-layer neural network with m ≥ n hidden units parametrized by Θ , as in Section 4.
Deﬁne the λ -regularized squared error loss
[Equation]
with Θ λ,m ∈ arg min Θ L λ,m (Θ) .
Suppose that equation D.3 is feasible with optimal solution α (cid:96) 1 .
Then as λ → 0 , L λ,m (Θ λ,m ) → 0 and (cid:107) Θ λ,m (cid:107) 22 → 2 (cid:107) α (cid:96) 1 (cid:107) 1 .
Proof.
We can see that equation D.3 will have a n -sparse solution α (cid:63) using the same reasoning as the proof of Lemma B.1.
Furthermore, following the proof of Proposition 4.1, the function x (cid:55)→ (cid:104) α (cid:63) , ϕ ( x ) (cid:105) is implementable by a neural network Θ (cid:63),m with (cid:107) Θ (cid:63) [Equation] m (cid:107) 22 = 2 (cid:107) α (cid:63) (cid:107) 1 = 2 (cid:107) α (cid:96) 1 (cid:107) 1 .
Following the same reasoning as before, we can also conclude that Θ (cid:63),m is an optimal solution for:
[Equation]
Now we note that λ (cid:107) Θ λ,m (cid:107) 22 ≤ L λ,m (Θ λ,m ) ≤ L λ,m (Θ (cid:63),m )
=
λ (cid:107) Θ (cid:63),m (cid:107) 22 ,
so as λ → 0 , and also (cid:107) Θ λ,m (cid:107) 2 ≤ (cid:107) Θ (cid:63),m (cid:107) 2 .
Now assume for the sake of contradiction that ∃ B with (cid:107) Θ λ,m (cid:107) 2 ≤ B < (cid:107) Θ (cid:63),m (cid:107) 2 for arbitrarily small λ .
We deﬁne
[Equation]
subject to (cid:107) Θ (cid:107) 2 ≤ B
Note that r (cid:63) > 0 since Θ (cid:63),m is optimal for equation D.5.
However, L λ,m ≥ r (cid:63) for arbitrarily small λ , a contradiction.
Thus, lim λ → 0 (cid:107) Θ (cid:63),m (cid:107) 22 = [Equation] Θ [Equation]
,m (cid:107) 22 .
We proceed to provide similar generalization bounds as the classiﬁcation setting.
This time, our bounds depend on the norms of the solution rather than the margin.
Let f φ ( α ; · ) (cid:44) x (cid:55)→ (cid:104) α, ϕ ( x ) (cid:105) (for ϕ ( x ) deﬁned in equation 4.1). Following the convention in Section C.2, deﬁne hypothesis class
F 1 ,φB (cid:44) { f φ ( α ; · ) : α ∈ L 1 ( S d − 1 ) , (cid:107) α (cid:107) 1 ≤ B } of linear functionals bounded by B in 1 -norm. As before, deﬁne F 2 ,φB and let ˆ R ( F ) denote the empirical Rademacher complexity of hypothesis class F .
We will ﬁrst derive a Rademacher complexity bound for F 1 ,φB . Claim D.6. ˆ R [Equation]
Proof. We write [Equation] We will now complete the bound on ˆ R ( F 1 ,φB ) for Lipschitz activations φ with φ (0) = 0 .
Claim D.7.
Suppose that our activation φ is M -Lipschitz and φ (0)
= 0 .Then
[Equation]
n
Proof.
We will show that
[Equation]
from which the statement of the lemma follows via Claim D
[Equation]
Fix any ¯ u (cid:48)
∈ S
d − 1 .
Then we get the decomposition
[Equation]
D.
[Equation]
We can bound the ﬁrst term as
[Equation]
since φ is Lipschitz and φ (0) = 0 )
[Equation]
We note that the second term of equation D.6 can be bounded by
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
This follows from the general fact that the difference between the supremum and inﬁmum of the absolute value of a quantity is bounded by the difference between the supremum and the inﬁmum.
Furthermore, by symmetry of the Rademacher random variables,
[Equation]
This simply gives an empirical Rademacher complexity of the hypothesis class F (cid:44) { x (cid:55)→ φ (¯ u (cid:62) x ) : ¯ u ∈ S d − 1 } scaled by n .
By the Lipschitz contraction property of Rademacher complexity, using the fact that φ is M -Lipschitz, we can therefore bound equation D.8 by
[Equation]
Plugging equation D.7 and equation D.9 back into equation D.6 gives the desired bound.
The following is a generalization bound based on the 1-norm:
Lemma D.8.
Let l ( · ; y ) :
R → [ − c, c ] be a bounded M -Lipschitz loss function.
Assume that φ is a 1-Lipschitz activation with φ (0) = 0 .
Let ( x i , y i ) ni
=1 be drawn i.i.d from p data .
Then with probability at least 1 − δ over the dataset, every α ∈ L 1 ( S d − 1 ) satisﬁes
[Equation]
2 n
Proof.
Our starting point is Theorem 1 of Kakade et al.
(2009), which states that with probability 1 − δ , for any ﬁxed hypothesis class F and f ∈ F ,
[Equation]
We deﬁne B j (cid:44) 2 j (cid:107) X (cid:107) F for j ≥ 0 .
We note that by Claim D.7, ˆ R ( F 1 ,φB j ) ≤ 3 2 j n . and apply the above on F 1 ,φB j using δ j (cid:44) δ 2 j +1 .
Then using a union bound, with probability 1 − (cid:80) ∞
j =
δ j = 1 − δ , for all j ≥ 0 and f φ ( α ; · ) ∈ F 1 ,φB j
[Equation]
n
Now for every α with (cid:107) α (cid:107) 1 < 1 (cid:107) X (cid:107) F , we use the inequality for F 1 ,φB 0 , and for every other α , we apply the inequality corresponding to F 1 ,φB j +1 , where 2 j ≤ (cid:107) α (cid:107) 1 (cid:107) X (cid:107) F ≤ 2 j +1 .
This gives the desired statement.
We can also provide the same generalization error bound for the 2-norm and relu features:
Lemma D.9.
In the setting of Lemma D.8, choose φ to be the relu activation.
Then with probability 1 − δ , every α ∈ L 2 ( S d − 1 ) satisﬁes
[Equation]
Proof.
We proceed the same way as in the proof of Lemma D.8.
We deﬁne B j as before, and this time have ˆ R ( F 2 ,φB j ) (cid:46) √ κ 2 j n √ d from Corollary C.4.
Thus, again union bounding over all j , equation equation D.10 gives with probability 1 − δ , for all j ≥ 0 and f φ ( α ; · ) ∈ F 2 ,φB j
[Equation]
n
Now we assign the α to different j as before to obtain the statement in the lemma.
Note that if l is some bounded loss such that l ( y ; y ) =
(for example, truncated squared error), for α (cid:96) 1 and α (cid:96) 2 the loss terms over the datapoints (in the bounds of Lemmas D.8 and D.9) vanish.
For loss l , deﬁne
[Equation]
Next, we will deﬁne the kernel matrix K with K ij = (cid:104) ϕ
( x i ) , ϕ ( x j ) (cid:105) .
Now we are ready to state and prove the formal theorem describing the gap between the 1-norm solution and 2-norm solution.
Theorem D.10.
Recall the deﬁnitions of α (cid:96) 1 and α (cid:96) 2 in equation D.3 and equation D.4.
For any activation φ with the property that K is full rank for any X with no repeated datapoints, there exists a distribution p data such that with probability 1,
[Equation]
On the other hand,
[Equation]
For i.i.d samples from this choice of p data , if l is bounded ( l ( · ; y ) :
R → [ − 1 , 1] ), 0 on correct predictions ( l ( y ; y ) = 0 ), and 1-Lipschitz, then with probability 1 − δ ,
[Equation]
n
Meanwhile, in the case that (cid:107) α (cid:96) 2 (cid:107) 22 ≥ nκ , the upper bound on L (cid:96) 2 -reg from Lemma D.9 is Ω(1) and in particular does not decrease with n .
We will ﬁrst show that for any dataset X , there is a distribution over Y such that the expectation of (cid:107) α (cid:96) 2 (cid:107) 2 is large.
When it is clear from context, y will denote the vector corresponding to Y .
Lemma D.11.
There is a distribution A over L 1 ( S d − 1 ) such that for any dataset X with y i (cid:44) (cid:104) ϕ
( x i ) , β (cid:105) for β ∼ A ,
[Equation]
and with probability 1,
[Equation]
We note the order of the quantiﬁers in Lemma D.11: the distribution A must not depend on the dataset X .
We ﬁrst provide a simple closed-form expression for (cid:107) α (cid:96) 2 (cid:107) 22 .
Claim D.12.
If K is full rank, then (cid:107) α (cid:96) 2 (cid:107)
= y (cid:62) K − 1 y .
Proof.
This follows by taking the dual of equation D.4.
Proof of Lemma D.11.
We sample β ∼ A as follows:
ﬁrst sample ¯ u ∼ S d − 1 uniformly.
Then set β to have a delta mass of 1 at ¯ u and be 0 everywhere else.
Deﬁne the vector v ¯ u (cid:44) [ φ (¯ u (cid:62) x 1 ) · · · φ (¯ u (cid:62) x n )] ; then it follows that we set our labels y to v ¯ u .
It is immediately clear that (cid:107) α (cid:96) 1 (cid:107) 1 ≤ (cid:107) β (cid:107) 1 ≤ 1 .
To lower bound E β [ (cid:107) α (cid:96) 2 (cid:107) 22 ] , from Claim D.12 we get E [Equation] 22 ] =
E
[Equation]
deﬁnition
[Equation]
κ Proof of Theorem D.10.
We note that since the distribution A of Lemma D.11 does not depend on the dataset X , it must follow that
[Equation]
[Equation] Thus, there exists β (cid:63) such that if we sample X i.i.d.
from the standard normal and set y i
=
(cid:104) ϕ
( x i ) , β (cid:63) (cid:105) , the expectation of (cid:107) α (cid:96) 2 (cid:107) 22 is at least nκ .
We choose p data corresponding to this β (cid:63) , with x sampled from the standard normal.
Now it is clear that p data will satisfy the norm conditions of
Theorem D.10.
For the generalization bounds, with high probability (cid:107) X (cid:107) F = Θ( √ nd ) as x is sampled from the standard normal distribution.
Thus, Lemma D.8 immediately gives the desired generalization error bounds for L (cid:96) 1 -reg .
On the other hand, if (cid:107) α (cid:96) 2 (cid:107) 2 ≥ (cid:112) nκ , then the bound of Lemma D.9 is at least √ κ (cid:107) [Equation] (cid:96) 2 (cid:107)
[Equation]
E M ISSING P ROOFS IN S ECTIONE. D ETAILED S ETUP
We ﬁrst write our regularity assumptions on Φ , R , and V in more detail:
Assumption E.
(Regularity conditions on Φ , R , V ) .
R is convex, nonnegative, Lipschitz, and smooth: ∃ M R , C R such that (cid:107)∇ 2 R (cid:107) op ≤ C R , and (cid:107)∇ R (cid:107) 2 ≤ M R .
Assumption E.2.
Φ is differentiable, bounded and Lipschitz on the sphere: ∃ B Φ , M Φ such that (cid:107) Φ(¯ θ ) (cid:107) ≤ B Φ ∀ ¯ θ ∈ S d , and | Φ i (¯ θ ) − Φ i (¯ θ (cid:48) )
| ≤ M Φ (cid:107) ¯ θ − ¯ θ (cid:48) (cid:107) 2 ∀ ¯ θ, ¯ θ (cid:48) ∈ S d .
Assumption E.3.
V is Lipschitz and upper and lower bounded on the sphere: ∃ b V , B V , M V such that 0 <
b V ≤ V (¯ θ ) ≤ B V ∀ ¯ θ ∈ S d , and (cid:107)∇ V (¯ θ ) (cid:107) 2 ≤ M V ∀ ¯ θ ∈ S d .
We state the version of Theorem 5.3 that collects these parameters:
Theorem E.4 (Theorem 5.3 with problem parameters) .
Suppose that Φ and V are 2-homogeneous and Assumptions E.1, E.2, and E.3 hold.
Fix a desired error threshold (cid:15) > 0 .
Suppose that from a starting distribution ρ 0 , a solution to the dynamics in equation 5.2 exists.
Choose
[Equation]
Then it must hold that min 0 ≤ t ≤ t (cid:15) L [ ρ t ] − inf ρ L [ ρ ] ≤ 2 (cid:15) .
E. P ROOF OF T HEOREM E.
Throughout the proof, it will be useful to keep track of W t (cid:44) (cid:112) E θ ∼ ρ t
[ (cid:107) θ (cid:107) 22 ] , the second moment of ρ t .
We ﬁrst introduce a general lemma on integrals over vector ﬁeld divergences.
Lemma E.5.
For any h 1 : R d +1 → R , h 2 : R d +1 → R d +1 and distribution ρ with ρ ( θ ) → 0 as (cid:107) θ (cid:107) → ∞ ,
[Equation]
Proof.
The proof follows from integration by parts.
We note that ρ t will satisfy the boundedness condition of Lemma E.5 during the course of our algorithm - ρ 0 starts with this property, and Lemma E.9 proves that ρ t will continue to have this property.
We therefore freely apply Lemma E.5 in the remaining proofs.
We ﬁrst bound the absolute value of L (cid:48)
[ ρ t ] over the sphere by B L (cid:44) M R B Φ + B V .
Lemma E.6.
For any ¯ θ ∈ S d − 1 , t ≥ 0 , | L (cid:48) [ ρ t ](¯ θ ) | ≤ (cid:44) B L .
Proof.
We compute
[Equation]
[Table]
[Equation]
Now we analyze the decrease in L [ ρ t ] .
Lemma E.7.
Under the perturbed Wasserstein gradient ﬂow
[Equation]
Proof.
Applying the chain rule, we can compute
[Equation]
where we use Lemma E.5 with h 1 = L (cid:48)
[ ρ t ] and h 2 = v [ ρ t ] .
Now we show that the decrease in objective value is approximately the average velocity of all parameters under ρ t plus some additional noise on the scale of σ .
At the end, we choose σ small enough so that the noise terms essentially do not matter.
Corollary E.8.
We can bound ddt L [ ρ t ] by
[Equation]
Proof.
By homogeneity, and Lemma
[Equation]
[Equation]
(¯ θ ) (cid:107) θ (cid:107) 22 ] ≤ B L W 2 t .
We also get E ¯ θ ∼ U
d
[ L (cid:48)
[ ρ t ](¯ θ )] ≤ B L since U d is only supported on S d .
Combining these with Lemma
E.7 gives the desired statement.
Now we show that if we run the dynamics for a short time, the second moment of ρ t will grow slowly, again at a rate that is roughly the scale of the noise σ .
Lemma E.9.
For all 0 ≤ t (cid:48) ≤ t , W 2 t (cid:48) ≤ L [ ρ 0 ]+ σtB L b V − tσB L .
Proof.
Let t ∗ (cid:44) arg max t (cid:48) ∈ [0 ,t ] W 2 t (cid:48) .
Integrating both sides of equation E.1, and rearranging, we get
[Equation]
Now since R is nonnegative, we apply L [ ρ t ∗ ] ≥ E θ ∼ ρ t ∗ [ V ( θ )] ≥ E θ ∼ ρ t ∗ [ V
[Equation]
¯ θ ) (cid:107) θ (cid:107) 22 ] ≥ b V W 2 t ∗ .
We now plug this in and rearrange to get W 2 t (cid:48) ≤ W 2 t ∗ ≤ L [ ρ 0 ]+ t ∗ σB L b V − t ∗ σB L ≤ L [ ρ 0 ]+ tσB L b V − tσB L ∀ 0 ≤ t (cid:48) ≤ t . Now let W 2 (cid:15) (cid:44) L [ ρ 0 ]+ σt (cid:15)
B L b v − t (cid:15) σB L .
By Lemma E.9, ∀ 0 ≤ t ≤ t (cid:15) , W 2 t ≤ W 2 (cid:15) .
The next statement allows us to argue that our dynamics will never increase the objective by too much.
Lemma E.10.
For any t 1 , t 2 with 0 ≤ t 1 ≤ t 2 ≤ t (cid:15) , L [ ρ t 2 ] − L [ ρ t 1 ] ≤ σ ( t 2 − t [Equation] B L ( W 2 (cid:15) + 1) .
Proof.
From Corollary E.8, ∀ t ∈
[ t 1 , t 2 ] we have
[Equation]
Integrating from t 1 to t 2 gives the desired result.
The following lemma bounds the change in expectation of a 2-homogeneous function over ρ t .
At a high level, we lower bound the decrease in our loss as a function of the change in this expectation.
Lemma E.11.
Let h : R d +
→ R that is 2-homogeneous, with (cid:107)∇ h (¯ θ ) (cid:107) ≤ M ∀ ¯ θ ∈ S d and | h (¯ θ ) | ≤ B ∀ ¯ θ ∈ S d .
Then ∀ 0 ≤ t ≤ t (cid:15) , we have
[Equation]
Proof.
Let Q ( t ) (cid:44) (cid:82) hdρ t .
We can compute:
[Equation]
Note that the ﬁrst two terms are bounded by σB ( W 2 (cid:15) + 1) by the assumptions for the lemma.
For the third term, we have from Lemma E.5:
[Equation]
[Equation]
since h is Lipschitz on the sphere)
[Equation]
Corollary [Equation]
Plugging this into equation E.3, we get that
[Equation]
We apply this result to bound the change in L (cid:48)
[ ρ t ] over time in terms of the change of the objective value.
For clarity, we write the bound in terms of c 1 that is some polynomial in the problem constants.
Lemma E.12.
Deﬁne Q ( t ) (cid:44) (cid:82) Φ dρ t .
For every ¯ θ ∈ S d and 0 ≤ t ≤ t + l ≤ t (cid:15) , ∃ c 1 (cid:44) poly ( k, C R , B Φ , M Φ , B L ) such that
[Equation]
Proof.
Recall that L (cid:48)
[ ρ t ](¯ θ ) =
(cid:104)∇ R
[Equation]
Φ dρ t ) , Φ(¯ θ ) (cid:105)
+ V (¯ θ ) .
Differentiating with respect to t ,
[Equation]
Integrating and applying the same reasoning to − L (cid:48)
[ ρ t ] gives us equation E.4.
Now we apply Lemma
E.11 to get
[Equation]
We plug this into equation E.6 and then integrate both sides to obtain
[Equation]
Using [Equation] B Φ M Φ , B L } gives the statement in the lemma.
Now we also show that L (cid:48) is Lipschitz on the unit ball.
For clarity, we let c 2 (cid:44) √ kM R M Φ + M V .
Lemma E.13.
For all ¯ θ, ¯ θ (cid:48) ∈ S d ,
[Equation]
Proof.
Using the deﬁnition of L (cid:48) and triangle inequality,
[Equation]
Now the remainder of the proof will proceed as follows: we show that if ρ t is far from optimality, either the expected velocity of θ under ρ t will be large in which case the loss decreases from
Corollary E.8, or there will exist ¯ θ such that L (cid:48)
[ ρ t ](¯ θ ) (cid:28) 0 .
We will ﬁrst show that in the latter case, the σU d noise term will grow mass exponentially fast in a descent direction until we make progress.
Deﬁne K −
[Equation]
t
[Equation] ≤ − τ } , the − τ -sublevel set of L (cid:48)
[ ρ t ] , and let m ( S ) (cid:44) E θ ∼ U d [ 1 ( θ ∈ S )] be the normalized spherical area of the set S .
Lemma E.14.
If K − τ t is nonempty, for 0 ≤ δ ≤ τ , log m ( K − τ + δ t ) ≥ − 2 d log c 2 δ .
Proof.
Let ¯ θ ∈ K − τ t .
From Lemma E.13, L (cid:48)
[ ρ ](¯ θ (cid:48) ) ≤ − τ + δ for all ¯ θ (cid:48) with (cid:107) ¯ θ (cid:48)
− ¯ θ (cid:107) 2 ≤ δc 2 .
Thus, we have
[Equation]
Now the statement follows by Lemma 2.3 of (Ball, 1997).
Now we show that if a descent direction exists, the added noise will ﬁnd it and our function value will decrease.
We start with a general lemma about the magnitude of the gradient of a 2-homogeneous function in the radial direction.
Lemma E.15.
Let h : R d +1 → R be a 2-homogeneous function.
Then for any θ ∈ R d +1 , ¯ θ (cid:62) ∇ h ( θ ) = 2 (cid:107) θ (cid:107) 2 h
(¯ θ ) .
Proof.
We have h ( θ + α ¯ θ ) =
( (cid:107) θ (cid:107) 2 + α ) 2 h
(¯ θ ) .
Differentiating both sides with respect to α and evaluating the derivative at 0, we get ¯ θ (cid:62) ∇ h ( θ ) = 2 (cid:107) θ (cid:107) 2 h
(¯ θ ) , as desired.
We state the lemma claiming that our objective will decrease if L (cid:48)
[ ρ t ](¯ θ ) (cid:28) 0 for some ¯ θ ∈ S d .
Lemma E.16.
Choose
[Equation]
If K − τ t ∗ is nonempty for some t ∗ satisfying t ∗ + l ≤ t (cid:15) , then after l steps, we will have
[Equation]
We will ﬁrst show that a descent direction in L (cid:48) [ ρ t ] will remain for the next l time steps.
In the notation of Lemma E.12, deﬁne z ( s ) (cid:44) C R B
Φ (cid:82) t ∗ + s t ∗ [Equation] Q (cid:48) ( t ) (cid:107) 1 dt .
Note that from Lemma E.12, for all ¯ θ
∈ S
d we have | L (cid:48)
[ ρ t ∗ + s ](¯ θ ) − L (cid:48)
[ ρ t ∗ ](¯ θ ) | ≤ z ( s ) .
Thus, the following holds:
Claim E.17.
For all s ≤ l , K − τ + z ( s ) t ∗ + s is nonempty.
Proof.
By assumption, ∃ ¯ θ with ¯ θ ∈ K − τ t ∗ .
Then L (cid:48)
[ ρ t ∗ + s ](¯ θ ) ≤ L (cid:48)
[ ρ t ∗ ](¯ θ )
+ z ( s ) ≤ − τ
+ z ( s ) , so
K − τ + z ( s ) t ∗ + s is nonempty.
Let T s (cid:44) K − τ/ 2+ z ( s ) t ∗
+ s for 0 ≤ s ≤ l .
We now argue that this set T s does not shrink as t increases.
Claim E.18.
For all s (cid:48) > s , T s (cid:48) ⊇ T s .
Proof.
From equation E.6 and the deﬁnition of z ( s ) , | L (cid:48)
[ ρ t + s (cid:48) ](¯ θ ) − L (cid:48)
[ ρ t + s ](¯ θ ) | ≤ z ( s (cid:48) )
− z ( s ) .
It follows that for ¯ θ ∈ T s
[Equation]
deﬁnition of T
[Equation]
[Equation]
which means that ¯ θ ∈ T s (cid:48) .
Now we show that the weight of the particles in T s grows very fast if z ( k ) is small.
Claim E.19.
Suppose that z ( l ) ≤ τ/ 4 .
Let ˜ T s = { θ ∈ R d
+1 : ¯ θ ∈ T s } .
Deﬁne N ( s ) (cid:44) (
cid:82) ˜ T s (cid:107) θ (cid:107) 2 dρ t ∗
+ s and β (cid:44) exp( − 2 d log 2 c 2 τ ) .
Then N (cid:48) ( s ) ≥ ( τ − σ ) N ( s ) + σβ .
Proof.
From the assumption z ( l ) ≤ τ 4 , it holds that
T s ⊆
K − τ/ 4 t ∗ + s ∀ s ≤ k .
Since T s is deﬁned as a sublevel set, v [ ρ t ∗ + s ](¯ θ ) points inwards on the boundary of T s for all ¯ θ ∈ T s , and by 1-homogeneity of the gradient, the same must hold for all u ∈ ˜ T s .
Now consider any particle θ ∈ ˜ T s .
We have that θ ﬂows to θ + v [ ρ t ∗ + s ]( θ ) ds at time t ∗
+ s + ds .
Furthermore, since the gradient points inwards from the boundary, it also follows that u + v [ ρ t ∗ + s ]( θ ) ds ∈ ˜ T s .
Now we compute
[Equation]
Now we apply Lemma E.15, using the 2-homogeneity of F (cid:48) and the fact that L (cid:48) [ ρ t ∗ + s ](¯ θ ) ≤ − τ/
[Equation]
Furthermore, since K − τ + z ( s ) t ∗ + s is nonempty by Claim E.17, we can apply Lemma E.14 and obtain
[Equation]
Plugging equation E.10 and equation E.11 back into equation E.9, we get
[Equation]
Since we also have that ˜ T s + ds ⊇ ˜ T s , it follows that
[Equation]
and so N (cid:48) ( s ) ≥ ( τ − σ ) N ( s ) + σβ .
Now we are ready to prove Lemma E.16.
Proof of Lemma E.16.
If z ( l ) =
C R B Φ (cid:82) t + l t (cid:107) Q (cid:48) ( t ) (cid:107) 1 ≥ τ 4 , then by rearranging the conclusion of
Lemma E.12 we immediately get equation E.8.
Suppose for the sake of contradiction that z ( l ) ≤ τ/ 4 .
From Claim E.19, it follows that N (1) ≥ σβ , and N ( l ) ≥ exp(( τ − σ )( l − 1))
N (1) .
Thus, in log( W 2 (cid:15) /σ )
+2 d log 2 c 2 τ τ − σ +1 time, W t ∗ + l ≥ N ( l ) ≥ W 2 (cid:15) , a contradiction.
Therefore, it must be true that z ( l ) ≥ τ/ 4 .
The following lemma will be useful in showing that the objective will decrease fast when ρ t is very suboptimal.
Lemma E.20.
For any time t with 0 ≤ t ≤ t (cid:15) , we have
[Equation]
Proof.
We can ﬁrst compute
[Equation]
via Lemma E.15)
[Equation] [Equation]
[Equation]
Rearranging gives
[Equation]
, and plugging this into equation E.1 gives the desired result.
Proof of Theorem E.4.
Let L (cid:63) denote the inﬁmum inf ρ L [ ρ ] , and let ρ (cid:63) be an (cid:15) -approximate global minimizer of L : L [ ρ (cid:63) ] ≤ L (cid:63) + (cid:15) .
(We deﬁne ρ (cid:63) because a true minimizer of L might not exist.)
Let
W (cid:63) (cid:44) E θ ∼ ρ (cid:63) [ (cid:107) θ (cid:107) 22 ] .
We ﬁrst note that since b V W (cid:63) 2 ≤ L [ ρ (cid:63) ] ≤ L [ ρ 0 ] , W (cid:63) 2 ≤ L [ ρ 0 ] /b
V ≤ W 2 (cid:15) .
Now we bound the suboptimality of ρ t : since L is convex in ρ ,
[Equation]
Rearranging gives
[Equation]
Now let l (cid:44)
[Equation]
(cid:1) , which satisﬁes Lemma E.16 with the value of τ later speciﬁed.
Suppose that there is a t with 0 ≤ t ≤ t (cid:15) − 2 l and ∀ t (cid:48) ∈
[ t,
[Equation]
+ 2 l ] , L [ ρ
[Equation]
] − L
[Equation]
≥ 2 (cid:15) .
Then L [ ρ t (cid:48) ] − L [ ρ (cid:63) ] ≥ (cid:15) .
We will argue that the objective decreases when we are (cid:15) suboptimal:
L [ ρ t ] − L [ ρ t +2 l ] ≥
[Equation] [Equation]
min
[Equation]
Using equation E.13 and W (cid:15) ≥ W (cid:63) , we ﬁrst note that
[Equation]
Thus, either min
[Equation]
such that the former holds, then the τ (cid:44)
[Equation]
W 2 (cid:15) sub-level set K − τ t (cid:48) is non-empty.
Applying Lemma
E.16 gives
[Equation]
Furthermore, from Lemma E.10, L [ ρ t +2 l ] − L [ ρ t (cid:48)
+ l ] ≤ σlc 1 ( W 2 (cid:15) + 1) and L [ ρ t (cid:48) ] − L [ ρ t ] ≤ σlB L ( W 2 (cid:15) + 1) , and so combining gives
[Equation]
In the second case E θ ∼ ρ t (cid:48)
[ L (cid:48)
[ ρ t (cid:48) ]
( θ )] ≥ (cid:15) 2 , ∀ t (cid:48) ∈
[ t, t + l ] .
Therefore, we can integrate equation E. from t to t + l in order to get
[Equation]
Therefore, applying Lemma E.10 again gives
[Equation]
Thus equation E.15 follows.
Now recall that we choose
[Equation]
For the simplicity, in the remaining computation, we will use O ( · ) notation to hide polynomials in the problem parameters besides d, (cid:15) .
We simply write σ = exp( − c 3 d log(1 /(cid:15) )) .
Recall our choice t (cid:15) (cid:44) O
( d 2 (cid:15) 4 log 2 (1 /(cid:15) )) .
It sufﬁces to show that our objective would have sufﬁciently decreased in t (cid:15) steps.
We ﬁrst note that with c 3 sufﬁciently large, W 2 (cid:15) =
O ( L [ ρ 0 ] /b v ) =
O (1) .
Simplifying our expression for l , we get that l = O ( d(cid:15) log 1 (cid:15) ) ,
so long as σW 2 (cid:15) =
o ( (cid:15) ) , which holds for sufﬁciently large c 3 .
Now let
[Equation]
Again, for sufﬁciently large c 3 , the terms with σ become negligible, and δ
=
O ( (cid:15) 2 l ) =
O ( (cid:15) 3 d log(1 /(cid:15) ) ) .
Likewise, δ 2 =
O ( d(cid:15) log(1 /(cid:15) )) .
Thus, if by time t we have not encountered 2 (cid:15) -optimal
ρ t , then we will decrease the objective by
O ( (cid:15) 3 d log(1 /(cid:15) ) ) in O ( d(cid:15) log 1 (cid:15) ) time.
Therefore, a total of O ( d 2 (cid:15) 4 log 2 (1 /(cid:15) )) time is sufﬁcient to obtain (cid:15) accuracy.
E. D ISCRETE -T IME O PTIMIZATION
To circumvent the technical issue of existence of a solution to the continuous-time dynamics, we also note that polynomial time convergence holds for discrete-time updates.
Theorem E.21.
Along with Assumptions E.
[Equation] E. [Equation] E.3 additionally assume that ∇ Φ i and ∇ V are
C Φ and C V -Lipschitz
[Equation]
respectively.
Let ρ t evolve according to the following discrete-time update:
[Equation]
There exists a choice of
[Equation]
such that min
[Equation]
The proof follows from a standard conversion of the continuous-time proof of Theorem E.4 to discrete time
[Equation]
and we omit it here for simplicity.
F A DDITIONAL M ATERIAL ON E XPERIMENTS F. D ETAILED S ETUP FOR F IGURE 2 E XPERIMENT
Our ground truth comes from a random neural network with 6 hidden units, and during training we use a network with as many hidden units as examples.
For classiﬁcation, we used rejection sampling to obtain datapoints with unnormalized margin of at least 0.1 on the ground truth network. We use a ﬁxed dimension of d = 20 . For all experiments, we train the network for 20000 steps with λ = 10 −and average over 100 trials for each plot point. F. D ETAILED S ETUP FOR F IGURE 3 E XPERIMENT
The left side of Figure 3 shows the experimental results for synthetic data generated from a ground truth network with 10 hidden units, input dimension d = 20 , and a ground truth unnormalized margin of at least 0 . We train for 80000 steps with learning rate 0 . 1 and λ = 10 − 5 , using two-layer networks with 2 i hidden units for i ranging from 4 to 10. We perform 20 trials per hidden layer size and plot the average over trials where the training error hit 0.
(At a hidden layer size of 2 7 or greater, all trials ﬁt the training data perfectly.)
The right side of Figure 3 demonstrates the same experiment, but performed on MNIST with hidden layer sizes of 2 i for i ranging from 6 to 15 .
We train forepochs using a learning rate of 0.01 and λ = 10 − 6 and use a single trial per plot point.
For MNIST, all trials ﬁt the training data perfectly.
The MNIST experiments are more noisy because we run one trial per plot point for MNIST, but the same trend of decreasing test error and increasing margin still holds.
F. V ERIFYING C ONVERGENCE TO THE M AX -M ARGIN
We verify the normalized margin convergence on a two-layer networks with one-dimensional input.
A single hidden unit computes the following: x (cid:55)→ a j relu ( w j x + b j ) .
We add (cid:107) · (cid:107) 22 -regularization to a, w , and b and compare the resulting normalized margin to that of an approximate solution of the (cid:96)
SVM problem with features relu ( wx i + b ) for w 2 + b 2 = 1 .
Writing this feature vector is intractable, so we solve an approximate version by choosing 1000 evenly spaced values of ( w, b ) .
Our theory predicts that with decreasing regularization, the margin of the neural network converges to the (cid:96)
SVM objective.
In Figure 4, we plot this margin convergence and visualize the ﬁnal networks and ground truth labels.
The network margin approaches the ideal one as λ → 0 , and the visualization shows that the network and (cid:96) 1 SVM functions are extremely similar.
[Table]
[Figure]
E XPERIMENTS ON CIFAR10 AND CIFAR
We train a modiﬁed WideResNet architecture (Zagoruyko & Komodakis, 2016) on CIFAR10 and
CIFAR100.
Our theory does not entirely apply because the identity mapping prevents ResNet architectures from being homogeneous, but our experiments show that reducing weight decay can still help generalization error in this setting.
Because batchnorm can cause the regularizer to have different effects (van Laarhoven, 2017), we remove batchnorm layers and train a 16 layer deep WideResNet.
We again compare a network trained with weight decayed annealing to one trained without annealing.
We used a ﬁxed learning rate schedule that starts at 0.1 and decreases by a factor of 0.2 at epochs 60, 120, and 160.
For CIFAR10, we use an initial weight decay of 0 . 0002 and decrease the weight decay by 0.2 at epoch 60, and then by 0.5 at epochs 90, 120, 140, 160.
For CIFAR100, we initialize weight decay at 0 . 0005 and decrease it by 0 .
2 at epochs 60, 120, and 160.
We tried different parameters for the initial weight decay and chose the ones that worked best for the model without annealing.
We also tried using small weight decays at initialization, but these models failed to generalize well – we believe this is due to an optimization issue where the algorithm fails to ﬁnd a true global minimum of the regularized loss.
We believe that annealing the weight decay directs the optimization algorithm closer towards the global minima for small λ .
Table 1 shows the test error achieved by models with and without annealing.
We see that the simple change of annealing