O N B REIMAN ’ S D ILEMMA IN N EURAL N ETWORKS : S UCCESS AND F AILURE OF N ORMALIZED M ARGINS
Anonymous authors
Paper under double-blind review
A BSTRACT
A belief persists long in machine learning that enlargement of margins over training data accounts for the resistance of models to overﬁtting by increasing the robustness.
Yet Breiman shows a dilemma (Breiman, 1999) that a uniform improvement on margin distribution does not necessarily reduces generalization error.
In this paper, we revisit Breiman’s dilemma in deep neural networks with recently proposed normalized margins using Lipschitz constant bound by spectral norm products.
With both simpliﬁed theory and extensive experiments, Breiman’s dilemma is shown to rely on dynamics of normalized margin distributions, that reﬂects the trade-off between model expression power and data complexity.
When the complexity of data is comparable to the model expression power in the sense that training and test data share similar phase transitions in normalized margin dynamics, Rademacher complexities of Lipschitz-normalized networks can be regarded as small constants and two efﬁcient ways are derived via classic marginbased generalization bounds to successfully predict the trend of generalization error.
On the other hand, over-expressed models that exhibit uniform improvements on training normalized margins may lose such a prediction power and fail to prevent the overﬁtting.
I NTRODUCTION
Margin, as a measurement of the robustness allowing some perturbations on classiﬁer without changing its decision on training data, has a long history in characterizing the performance of classiﬁcation algorithms in machine learning.
As early as Novikoff (1962), it played a central role in the proof on ﬁnite-stopping or convergence of perceptron algorithm when training data is separable.
Equipped with convex optimization technique, a plethora of large margin classiﬁers are triggered by support vector machines (Cortes & Vapnik, 1995; Vapnik, 1998).
AdaBoost, an iterative algorithm to combine an ensemble of classiﬁers proposed by Freund & Schapire (1997), often exhibits a resistance to overﬁtting phenomenon that during the training process the generalization error keeps on non-increasing when the training error drops to zero.
Toward deciphering the such a resistance of overﬁtting phenomenon, Schapire et al.
(1998) proposed an explanation that the training process keeps on improving a notion of classiﬁcation margins in boosting, among later works on consistency of boosting with early stopping regularization (B¨uhlmann & Yu, 2002; Zhang & Yu, 2005;
Yao et al., 2007).
Lately such a resistance to overﬁtting is again observed in deep neural networks with overparameterized models (Zhang et al., 2016).
A renaissance of margin theory is proposed by
Bartlett et al.
(2017) with a normalization of network using Lipschitz constants bounded by products of operator spectral norms.
It inspires many further investigations in various settings (Miyato et al., 2018; Neyshabur et al., 2018; Liao et al., 2018).
However, the improvement of margin distributions does not necessarily guarantee a better generalization performance, which is at least traced back to (Breiman, 1999) in his effort to understanding
AdaBoost.
In this work, Breiman designed an algorithm arc-gv such that the margin can be maximized via a prediction game, then he demonstrated an example that one can achieve uniformly larger margin distributions on training data than AdaBoost but suffer a higher generalization error.
In the end of this paper, Breiman made the following comments with a dilemma: ”The results above leave us in a quandary.
The laboratory results for various arcing algorithms are excellent, but the theory is in disarray.
The evidence is that if we try too hard to make the margins larger, then overﬁtting sets in. ...
My sense of it is that we just do not understand enough about what is going on.”
Breiman’s dilemma triggers some further explorations to understand the limitation of margin theory in boosting (Reyzin & Schapire, 2006; Wang et al., 2008; 2011).
In particular, Reyzin & Schapire (2006) points out that the trees found by arg-gv have larger model complexity in terms of deeper average depth than AdaBoost, suggesting that margin maximization in arc-gv does not necessarily control the model complexity.
The latter works provide tighter bounds based on VC-dimension and optimized quantile training margins, which however do not apply to over-parametrized models in deep neural networks and the case where the training margin distributions are uniformly improved.
In this paper, we are going to revisit Breiman’s dilemma in the scenario of deep neural networks.
Both the success and failure can be seen on normalized margin based bounds on generalization error.
First of all, let’s look at the following illustration example.
Example (Breiman’s Dilemma with a CNN) .
A basic 5-layer convolutional neural network of c channels (see Section 3 for details) is trained with CIFAR-10 dataset whose 10 percent labels are randomly permuted.
When c = 50 with 92 , 610 parameters, Figure 1 (a) shows the training error and generalization (test) error in solid curves.
From the generalization error in (a) one can see that overﬁtting indeed happens after about 10 epochs, despite that training error continuously drops down to zero.
One can successfully predict such an overﬁtting phenomenon from Figure 1 (b), the evolution of normalized margin distributions deﬁned later in this paper.
In (b), while small margins are monotonically improved during training, large margins undergoes a phase transition from increase to decrease around 10 epochs such that one can predict the tendency of generalization error in (a) using large margin dynamics.
Two particular sections of large margin dynamics are highlighted in (b), one at 8.3 on x -axis that measures the percentage of normalized training margins no more than 8.3 (training margin error) and the other at 0.8 on y -axis that measures the normalized margins at quantile q = 0 .
8 (i.e. 1 / ˆ γ q,t ).
Both of them meet the tendency of generalization error in (a) and ﬁnd good early stopping time to avoid overﬁtting.
However, as we increase the channel number to c = 400 with about 5 .
8 M parameters and retrain the model, (c) shows a similar overﬁtting phenomenon in generalization error; on the other hand, (d) exhibits a monotonic improvement of normalized margin distributions without a phase transition during the training and thus fails to capture the overﬁtting.
This demonstrates the Breiman’s dilemma in CNN.
[Figure]
A key insight behind this dilemma, is that one needs a trade-off between the model expression power and the complexity of the dataset to endorse margin bounds a prediction power.
On one hand, when the model has a limited expression power relative to the training dataset, in the sense that the training margin distributions CAN NOT be uniformly improved during training, the generalization or test error may be predicted from dynamics of normalized margin distributions.
On the other hand, if we push too hard to improve the margin by giving model too much degree of freedom such that the training margins are uniformly improved during training process, the predictability may be lost.
A trade-off is thus necessary to balance the complexity of model and dataset, otherwise one is doomed to meet Breiman’s dilemma when the models arbitrarily increase the expression power.
The example above shows that the expression power of models relative to the complexity of dataset, can be observed from the dynamics of normalized margins in training, instead of counting the number of parameters in neural networks.
In the sequel, our main contributions are to make these precise by revisiting the Rademacher complexity bounds with Lipschitz constants (Bartlett et al., 2017).
• With the Lipschitz-normalized margins, a linear inequality is established between training margin and test margin in Theorem 1.
When both training and test normalized margin distributions undergo similar phase transitions on increase-decrease during the training process, one may predict the generalization error based on the training margins as illustrated in Figure 1.
• In a dual direction, one can deﬁne a quantile margin via the inverse of margin distribution functions, to establish another linear inequality between the inverse quantile margins and the test margins as shown in Theorem 2.
Quantile margin is far easier to tune in practice and enjoys a stronger prediction power exploiting an adaptive selection of margins along model training.
In all cases, Breiman’s dilemma may fail both of the methods above when dynamics of normalized training margins undergo different phase transitions to that of test margins during training, where a uniform improvement of margins results in overﬁtting.
Section 2 describes our method to derive the two linear inequalities of generalization bounds above.
Extensive experimental results are shown in Section 3 and Appendix with basic CNNs, AlexNet,
VGG, ResNet, and various datasets including CIFAR10, CIFAR100, and mini-Imagenet.
M ETHOD
Let X be the input space (e.g. X ⊂ R C × W × H in image classiﬁcation) and Y :
= { 1 , . . .
, K } be the space of K classes.
Consider a sample set of n observations S
= { ( x 1 , y 1 ) , . . .
, ( x n , y n ) : x i ∈
X , y i ∈ Y} that are drawn i.i.d.
from P X,Y .
For any function f : X → R , let P f = (cid:82) X f ( X ) dP be the population expectation and P n f = (1 /n ) (cid:80)
ni
=1 f ( x i ) be the sample average.
Deﬁne F to be the space of functions represented by neural networks,
[Equation]
where l is the depth of the network, W i is the weight matrix corresponding to a linear operator on x i and σ i stands for either element-wise activation function (e.g. ReLU) or pooling operator that are assumed to be Lipschitz bounded with constant L σ i and satisfying σ i (0)
= 0 .
For example, in convolutional network, W i x i + b i = w i ∗ x
i + b i where ∗ stands for the convolution between input tensor x l and kernel tensor w l .
We equip F with the Lipschitz semi-norm, for each f ,
[Equation]
where (cid:107) · (cid:107) σ is the spectral norm and L σ = (cid:81)
Li
=
1 L σ i .
For all the examples in this paper, we use
ReLU activation σ i that leads to L σ i = 1 .
Moreover we consider the following family of hypothesis mapping,
[Equation]
where [ · ] j denotes the j th coordinate and we further deﬁne the following class induced by Lipschitz semi-norm bound on F ,
[Equation]
Lastly, rather than merely looking at whether a prediction f ( x ) on y is correct or not, we also consider the margin deﬁned as ζ ( f ( x ) , y ) =
[ f ( x )]
y − max { j : j (cid:54) =
y } [ f ( x )]
j .
Therefore, we can deﬁne the ramp loss and margin error depending on the conﬁdence of predictions.
Given two thresholds γ 2 > γ 1 ≥ 0 , deﬁne a ramp loss to be
[Equation]
where ∆ := γ 2 − γ 1 .
In particular γ 1 = 0 and γ
= γ , we also write (cid:96) γ = (cid:96) γ for simplicity.
Deﬁne the margin error to measure if f has margin no more than a threshold γ ,
[Equation]
In particular, e 0 ( f ( x ) , y ) is the common mis-classiﬁcation error and E [ e 0 ( f ( x ) , y )]
=
P [ ζ ( f ( x ) , y ) < 0] .
Note that e 0 ≤ (cid:96) γ ≤ e γ , and (cid:96) γ is Lipschitz bounded by 1 /γ .
The central question we try to answer is, can we ﬁnd a proper upper bound to predict the tendency of the generalization error along training, such that one can early stop the training near the epoch that P [ ζ ( f t ( x ) , y ) < 0] is minimized?
The answer is both a yes and a no !
We begin with the following lemma, as a typical result in multi-label classiﬁcation from the uniform law of large numbers (Koltchinskii et al., 2002).
Lemma 2.1.
Given a γ 0 > 0 , then, for any δ ∈ (0 , 1) , with probability at least 1 − δ , the following holds for any f ∈ F with (cid:107) f (cid:107) F ≤ L ,
[Equation]
where
[Equation]
is the Rademacher complexity of function class H L with respect to n samples, and the expectation is taken over x i , ε i , i = 1 , ..., n .
Unfortunately, direct application of such bound for a constant γ 0 will suffer from the so-called scaling problem .
The following proposition gives an lower bound of Rademacher complexity term, whose proof is provided in Appendix D.
Proposition 1. Consider the networks with ReLU activation functions.
For any L > 0 , there holds,
[Equation]
where C > 0 is a constant that does not depend on S .
The lemma tells us if L → ∞ , upper bound (6) becomes trivial since R n ( H L ) → ∞ .
In fact, both
Telgarsky (2013) and Soudry et al.
(2018) show that with gradient descent, the norm of estimator’s weight in logistic regression and general boosting (including exponential loss), respectively, will go to inﬁnity at a growth rate log( t ) when the data is linearly separable.
As for the deep neural network with cross-entropy loss, the input of last layer is usually be viewed as features extracted from original input.
Training the last layer with other layers ﬁxed is exactly a logistic regression, and the feature is linearly separable as long as the training error achieves zero.
Therefore, without any normalization, the hypothesis space along training has no upper bound on L and the upper bound (6) is useless.
Besides, even for a ﬁxed L , the complexity term R n ( H L ) is computationally intractable.
The ﬁrst remedy is to restrict our attention on H 1 by normalizing f with its Lipschitz semi-norm (cid:107) f (cid:107) F or its upper bounds.
Note that a normalized network ˜ f
= f/C has the same mis-classiﬁcation error as f for all C > 0 .
For the choice of C , it’s hard in practice to directly compute the Lipschitz semi-norm of a network, but instead some approximate estimates on the upper bound L f in (2) are available as discussed in Appendix A.
In the sequel, let ˜ f
= f/L f be the normalized network and
˜ h
= h/L f = ζ ( f, y ) /L
f = ζ ( ˜ f, y ) ∈ H 1 be the corresponding normalized hypothesis function.
Now a simple idea is to regard R n ( H 1 ) as a constant and predict the tendency of generalization error via training margin error of the normalized network, that avoids the scaling problem and the computation of complexity term.
The following theorem makes this precise.
Theorem 1.
Given γ 1 and γ 2 such that γ 2 > γ 1 ≥ 0 and ∆ := γ 2 − γ 1 ≥ 0 , for any δ > 0 , with probability at least 1 − δ , along the training epoch t = 1 , . . .
, T , the following holds for each f t ,
[Equation]
where C H = 2 K 2 R n ( H 1 ) .
Remark.
In particular, when we take γ 1 = 0 and γ 2 = γ > 0 , the bound above becomes,
[Equation]
Theorem 1 says, we can bound the normalized test margin distribution P [ ζ [Equation] ˜ f [Equation]
x [Equation] , y ) < γ 1 ] by the normalized training margin distribution P n [ ζ ( ˜ f t ( x ) , y ) <
γ 2 ] .
Recently Liao et al.
(2018) investigates for normalized networks, the strong linear relationship between cross-entropy training loss and test loss when the training epochs are large enough.
As a contrast, we consider the whole training process and normalized margins.
In particular, we hope to predict the trend of generalization error by choosing γ 1 = 0 and a proper γ .
For this purpose, the following facts are important.
First, we do not expect the bound, for example (10), is tight for every choice of γ > 0 , instead we hope there exists some γ such that the training margin error nearly monotonically changes with generalization error.
Figure 2 shows the existence of such γ such that the training margin error successfully recover the tendency of generalization error on CIFAR10 dataset.
Moreover, in Appendix Figure 8 shows the rank correlation between training margin error at various γ and training/test error.
Second, the normalizing factor is not necessarily to be an upper bound of Lipschitz semi-norm.
The key point is to prevent the complexity term of the normalized network going to inﬁnity.
Since for any constant c > 0 , normalization by ¯ L = cL works in practice where the constant could be absorbed to γ , we could ignore the Lipschitz constant introduced by general activation functions in the middle layers.
However, it is a natural question whether a reasonable γ with prediction power exists.
A simple example in Figure 1 shows, once the training margin distribution is uniformly improved, dynamic of training margin error fails to detect the minimum of generalization error in the early stage.
This is because when network structure becomes complex enough, the training margin distribution could be more easily improved but the the generalization error may overﬁt.
This is exactly the same observation in Breiman (1999) to doubt the margin theory in boosting type algorithms.
More detailed discussions will be given in Section 3.2.
The most serious limitation of Theorem 1 lies in we must ﬁx a γ along the complete training process.
In fact, the ﬁrst term and second term in the bound (10) vary in the opposite directions with respect to γ , and thus different f t may prefer different γ for a trade-off.
As in Figure 1 (b) of the example, while choosing γ is to ﬁx an x -coordinate section of margin distributions, its dual is to look for a y -section which leads to different margins for different f t .
This motivates the quantile margin in the following theorem.
Let ˆ γ q,f be the q th quantile margin of the network f with respect to sample S ,
[Equation]
Theorem 2.
Assume the input space is bounded by M > 0 , that is (cid:107) x (cid:107) 2 ≤ M, ∀ x ∈ X .
Given a quantile q ∈ [0 , 1] , for any δ ∈ (0 , 1) and τ > 0 , the following holds with probability at least 1 − δ for all f t satisfying ˆ γ q, ˜ f t > τ ,
[Equation]
Remark.
We simply denote γ q,t for γ q, ˜ f t when there is no confusion.
Compared with the bound (10), (12) make the choice of γ varying with f t and the cost is an additional constant term C 2 q and the constraint ˆ γ q,t > τ that typically holds for large enough q in practice.
In applications, stochastic gradient descent (SGD) often effectively improves the training margin distributions along the drops of training errors, a small enough τ and large enough q usually meet ˆ γ q,t > τ .
Moreover, even with the choice τ = exp( − B ) , constant term (cid:112) [log log 2 (4( M + l ) /τ )]
/n =
O ( (cid:112) log B/n ) is still negligible and thus very little cost is paid in the upper bound.
In practice, tuning q ∈ [0 , 1] is far easier than tuning γ > 0 directly and setting a large enough q ≥ 0 .
9 usually provides us lots of information about the generalization performance.
The quantile margin works effectively when the dynamics of large margin distributions reﬂects the behavior of generalization error, e.g. Figure 1.
In this case, after certain epochs of training, the large margins have to be sacriﬁced to further improve small margins to reduce the training loss, that typically indicates a possible saturation or overﬁtting in test error.
E XPERIMENTAL R ESULTS
We brieﬂy introduce the network and dataset used in the experiments.
For the network, we ﬁrst consider the convolutional neural network with very simple structure basic CNN( c ) .
The structure is shown in Appendix Figure 7.
Basically, it has ﬁve convolutional layers with c channels at each and one fully connected layer, where c will be speciﬁed in concrete examples.
Second, we consider more practical network structure, AlexNet
(Krizhevsky et al., 2012), VGGNet-16 (Simonyan & Zisserman, 2014) and ResNet-18 (He et al., 2016).
For the dataset, we consider CIFAR10, CIFAR (Krizhevsky & Hinton, 2009) and Mini-ImageNet (Vinyals et al., 2016).
The spirit of the following experiments is to show, when and how, the margin bound could be used to predict the tendency of generalization or test error along the training path?
3. S UCCESS : T RAINING M ARGIN E RROR AND Q UANTILE M ARGIN
This section is to apply Theorem 1 and Theorem 2 to predict the tendency of generalization error.
Let’s ﬁrstly consider training a basic CNN(50) on CIFAR10 dataset with and without random noise.
The relations between generalization error and training margin error e γ ( ˜ f
( x ) , y ) with γ = 9 . 8 , inverse quantile margin 1 / ˆ γ q,t with q = 0 .
6 are shown in Figure 2.
In this simple example where the net is light and the dataset is simple, the linear bounds (9) and (12) show a good prediction power: they stop either near the epoch of sufﬁcient training (Left, original data) or where even an overﬁtting occurs (Right, 10 percents label corrupted).
A few discussions are given below.
There exists a trade-off on the choice of γ from the linear bounds (9) (and parallel arguments hold for q ).
The training margin error with a small γ is close to the training error, while a large γ is close to generalization error and it’s illustrated in Appendix Figurewhere we show the Spearman’s ρ rank correlation 1 between training margin error and training error, generalization error against threshold γ .
The training margin error (or inverse quantile margin) is closely related to the dynamics of training margin distributions.
For certain choice of γ , if the curve of training margin error (with respect to epoch) is V-shape, the corresponding dynamics of training margin distributions will have a cross-over , where the low margins have a monotonic increase and the large margins undergo a phase transition from increase to decrease, as illustrated by the red arrow in Figure 1 (b).
Dynamics of quantile margins can adaptively select γ t for each f t without access to the complexity term.
Unlike merely looking at the training margin error with a ﬁxed γ , quantile margin bound (12) shows a stronger prediction power than (10) and even be able to capture more local information as illustrated in Figure 3.
The generalization error curve has two valleys corresponding to a local optimum and a global optimum, and the quantile margin curve with q = 0 .
95 successfully identiﬁes both.
However, if we consider the dynamics of training margin errors, it’s rarely possible to recover the two valleys at the same time since their critical thresholds γ t 1 and γ t 2 are different.
Another example of ResNet is given in
Appendix Figure 9.
3. F AILURE : B REIMAN ’ S D ILEMMA IN O VER -P ARAMETERIZED M ODELS
In this section, we explore the normalized margin dynamics with over-parameterized models whose expression power might be greater than data complexity.
We conduct experiments in the following two scenarios.
In the ﬁrst experiment shown in Figure 4, we ﬁx the dataset to be CIFAR10 with 10 percent of labels randomly permuted, and gradually increase the channels from basic CNN(50) to basic CNN(400).
As the channel number increases, dynamics of the normalized training margins in the ﬁrst row change from a phase transition with a cross-over in large margins to a monotone improvement of margin distributions.
This phenomenon is not a surprise since with a strong representation power, the whole training margin distribution can be monotonically improved without sacriﬁcing the large margins.
On the other hand, the generalization or test error can never be monotonically improved.
In the second row, heatmaps depict rank correlations of dynamics between training and test margin errors, which clearly show the phase transitions for CNN(50) and CNN(100) and its disappearance for CNN(400).
In the second experiment shown in 5, we compare the normalized margin dynamics of training CNN(400) and ResNet18 on two different datasets, CIFAR100 (the simpler) and
Mini-ImageNet (the more complex).
It shows that: (a) CNN(400) (5.8M parameters) does not have an over-representation power on CIFAR100, whose normalized training margin dynamics exhibits a phase transition – a sacriﬁce of large margins to improve small margins during training; (b) ResNet18 (11M parameters) exhibits an over-representation power on CIFAR100 via a monotone improvement on training margins, but loses such a power in Mini-ImageNet with the phase transitions in margin dynamics
More experiments including AlexNet and VGG16 are shown in Appendix Figure 11.
This phenomenon is not unfamiliar to us, since Breiman (Breiman, 1999) has pointed out that the improvement of training margins is not enough to guarantee a small generalization or test error in the boosting type algorithms.
In this paper Breiman designed an algorithm, called arc-gv , enjoying an uniformly better training margin distribution comparing with Adaboost but suffer a higher generalization error.
Now again we ﬁnd the same phenomenon ubiquitous in deep neural networks we further explore
how
In the end, it’s worth mentioning different choices of the normalization factor estimates may affect the range of predictability.
In all experiments above, normalization factor is estimated via an upper bound on spectral norm given in Appendix A (Lemma A.1 in Section A).
One could also use power iteration (Miyato et al., 2018) to present a more precise estimation on spectral norm.
It turns out a more accurate estimation of spectral norm can extend the range of predictability, but Breiman’s dilemma is still there when the balance between model expression power and dataset complexity is broken.
More experiments on this aspect can be found in Figure 10 in Appendix.
C ONCLUSION
In this paper we show that Breiman’s dilemma is ubiquitous in deep learning, in addition to previous studies on Boosting algorithms.
We exhibit that Breiman’s dilemma is closely related to the tradeoff between model expression power and data complexity.
A novel perspective on phase transitions in dynamics of Lipschitz-normalized margin distributions is proposed to inspect when the model has over-representation power compared to the dataset, instead of merely counting the number of parameters.
A data-driven early stopping rule by monitoring the margin dynamics is a future direction to explore.
Lipschitz semi-norm plays an important role in normalizing or regularizing neural networks, e g. in GANs (Kodali et al., 2017; Miyato et al., 2018), therefore a more careful treatment deserves further pursuits
A E STIMATE OF N ORMALIZATION F ACTORS
In this section we discuss how to estimate the Lipschitz constant bound in (2).
Given an operator
W associated with a convolutional kernel w , i.e. Wx = w ∗ x , there are two ways to estimate its operator norm.
We begin with a useful lemma,
Lemma A.1.
For convolution operator with kernel w , i.e. Wx :=
w ∗ x , there holds (cid:107)
[Equation]
In other words, (cid:107) W (cid:107) σ ≤ (cid:107) w (cid:107) 1 .
Proof.
[Equation]
where the second last step is due to Cauchy-Schwartz inequality.
A. (cid:96) 1 -norm.
The convolutional operator (spectral) norm can be upper bounded by the (cid:96) 1 -norm of its kernels, i.e. (cid:107) W (cid:107) σ ≤ (cid:107) w (cid:107) 1 .
This is a simple way but the bound gets loose when the channel numbers increase.
B. Power iteration.
A fast approximation for the spectral norm of the operator matrix is given in (Miyato et al., 2018) in GANs that is based on power
iterations (Golub & Van der Vorst, 2001).
Yet as a shortcoming, it is not easy to apply to the ResNets.
We compare two estimation in Appendix Figure 10.
It turns out both of them have prediction power on the tendency of generalization error and both of them will fail when the network has large enough expression power.
Though using (cid:96) 1 norm of kernel is extremely efﬁcient, the power iteration method may be tighter and has a wider range of predictability.
In the remaining of this section, we will particularly discuss the treatment of ResNets.
ResNet is usually a composition of the basic blocks shown in Figure 6 with short-cut structure.
The following method is used in this paper to estimate the upper bound of operator or spectral norm of such a basic block of ResNet.
(a) Convolution layer: its operator norm can be bounded either by the (cid:96) 1 norm of kernel or by power iteration above.
(b) Batch Normalization (BN): in training process, BN normalizes samples by x + = ( x − µ B ) / (cid:112) σ 2 B + (cid:15) , where µ B , σ 2 B are mean and variance of batch samples, while keeping an online averaging as ˆ µ and ˆ σ 2 .
Then BN rescales x + by estimated parameters
ˆ α, ˆ β and output
ˆ x = ˆ αx + + ˆ β .
Therefore the whole rescaling of BN on the kernel tensor w
of the convolution layer
is ˆ w = w ˆ α/ √ ˆ σ 2 + (cid:15) and its corresponding rescaled operator is (cid:107) ˆ W (cid:107) σ = (cid:107) W (cid:107) σ ˆ α/ √ ˆ σ 2 + (cid:15) .
(b) Activation and pooling: their Lipschitz constants L σ can be known a priori, e.g. L σ =for ReLU and hence can be ignored.
In general, L σ can not be ignored if they are in the shortcut as discussed below.
(d) Shortcut: In residue net with basic block in Figure 6, one has to treat the mainstream ( Block 2 , Block 3 ) and the shortcut Block 1 separately.
Since (cid:107) f + g (cid:107) F ≤ (cid:107) f
(cid:107) F +
(cid:107) g (cid:107) F , in this paper we take the Lipschitz upper bound by L σ out ( (cid:107) ˆ W 1 (cid:107) σ + L σ in (cid:107) ˆ W 2 (cid:107) σ (cid:107) ˆ W 3 (cid:107) σ ) , where (cid:107)
ˆ W i (cid:107) σ denotes a spectral norm estimate of BN-rescaled convolutional operator W i .
In particular L σ out can be ignored since all paths are normalized by the same constant while
L σ in can not be ignored due to its asymmetry.
B S TRUCTURE OF B ASIC CNN
C E XPERIMENTS
[Figure] [Figure]
The picture is slight different here, since after the ﬁrst (better) local minimum, the training margin distribution is uniformly improved without reducing generalization error.
Therefore, we could not expect the inverse quantile margin to reﬂect the tendency of generalization error globally, especially the order of two local minimums.
However, around epochs when local minimum occurs, the training margin distribution still has a cross-over, and thus the inverse quantile margin could reﬂect the tendency locally
C. E STIMATED BY POWER ITERATION AND KERNEL l 1 NORM
C. S UCCESS AND FAILURE IN MORE PRACTICAL NETWORK AND DATASET
D P ROOFS D. A UXILIARY L EMMAS
Lemma 2.1 follows from the following Lemma D.1 by applying it to bounded function (cid:96) γ ( f ( x ) , y ) whose range is [0 , 1] .
Lemma D.1.
For any δ ∈ (0 , 1) and bounded-value functions F B := { f : X → R : (cid:107)
[Equation]
∞ ≤ B } , the following holds with probability at least 1 − δ ,
[Equation]
) where
[Equation]
is the Rademacher Complexity of function class F .
For completeness, we include its proof that also needs the following well-known McDiarmid’s inequality (see, e.g. Wainwright (2019)).
Lemma D.2 (McDiarmid’s Bounded Difference Inequality) .
For B i -bounded difference functions h :
[Equation]
t
[Equation]
i
[Equation]
Proof of Lemma D.1.
It sufﬁces to show that for ¯ f = f ( x [Equation] − E f
[Equation]
[Equation]
where with probability at least 1 − δ ,
[Equation]
by McDiarmid’s bounded difference inequality, and
[Equation]
using Rademacher complexity.
To see (16), we are going to show that sup f ∈F B E n ¯ f is a bounded difference function.
Consider g ( x n 1 ) =
E n ¯ f = 1 n (cid:80) ni
=1 f ( x i ) − E x f ( x ) .
Assume that the i -th argument x i changes to x (cid:48) i , then for every g ,
[Equation]
Hence sup g g ( x i , x − i ) − sup g g ( x (cid:48)
i , x − i ) ≤ B/n , which implies that sup f ∈F B E n ¯ f is a B/nbounded difference function.
Then (16) follows from the McDiarmid’s inequality (Lemma D.2) using B i
=
B/n and δ = exp( − 2 nε 2 /B 2 ) .
As to (17),
[Equation]
that ends the proof.
We also need the following contraction inequality of Rademacher Complexity (Ledoux & Talagrand, 1991; Meir & Zhang, 2003).
Lemma D.3 (Rademacher Contraction Inequality) .
For any Lipschitz function: φ : R → R such that | φ ( x ) − φ ( y ) | ≤ L | x − y | ,
[Equation]
Ledoux & Talagrand (1991) has an additional factor 2 in the contraction inequality which is dropped in Meir & Zhang (2003).
Its current form is stated in Mohri et al.
(2012) as Talagrand’s
Lemma (Lemma 4.2).
Beyond, we further introduce the family,
[Equation]
and the sub-family constraint in Lipschitz semi-norm on f
[Equation]
[Equation] The following lemma (Koltchinskii et al., 2002) allows us to bound the Rademacher complexity term of R n ( G ) by R n ( H ) ,
Lemma D.4.
R n ( G L ) ≤ K 2 R n ( H L )
Proof of Lemma D.4.
R
[Equation]
n
[Equation]
(cid:107)
[Equation]
(cid:107)≤ L n (cid:88) i = (cid:15) {+i ζ ( f ( x i ) , y i ) , =n E S,(cid:15) sup (cid:107) f (cid:107)≤ L n (cid:88) i = (cid:88) y ∈Y (cid:15) i ζ ( f ( x i ) , y i )
+}
[ y i = y ] , =n (cid:88)
y ∈Y
E S,(cid:15) sup (cid:107) f
(cid:107)≤ L n (cid:88) i = (cid:15) i ζ ( f ( x i {+) , y i )
 [ y i =+}
y ] , =n (cid:88) y ∈Y E S,(cid:15) [ sup (cid:107) f (cid:107)≤ L n (cid:88) i = {+(cid:15) i (2 · 
+}
[ y i =
y ] − +2)] , ≤2 n (cid:88)
y ∈Y
E S,(cid:15) [ sup (cid:107) f
(cid:107)≤ L n (cid:88) i = (cid:15) i {+(2 · 
+}
[ y i = y ] − 1) ]
+ 1 2 n (cid:88)
y ∈Y
E S,(cid:15) [ sup (cid:107) f
(cid:107)≤ L m (cid:88) i = (cid:15) i ζ ( f ( x i ) , y )]) , =n (cid:88) y ∈Y E S,(cid:15) [ sup (cid:107) f [Equation]
L [Equation] i = [Equation]
( f ( x i [Equation] [Equation] , =n
[Equation]
max y (cid:48) (cid:54) [Equation] [Equation] (cid:48)
[Equation]
, ≤n (cid:88) [Equation] L
[Equation]
i = [Equation] [Equation] (cid:88) y ∈Y
E S [Equation]
(cid:15) [ sup (cid:107) f
(cid:107)≤ L m (cid:88) i = (cid:15) i max y (cid:48)
(cid:54) =
y [ f ( x i )] y (cid:48) ] , [Equation] L
[Equation]
i
[Equation]
[Equation] K [Equation] R n ( H L ) , where the last inequality is implied from R n ( { max( f 1 , . . .
, f M ) :
f i ∈ F i } ) ≤ (cid:80)
Mm =1 R n ( F m ) (Koltchinskii et al., 2002;
Mohri et al., 2012).
D. P ROOF OF P ROPOSITION
Proof of Proposition 1.
Without loss of generality, we assume L σ i
= 1 , i = 1 , . . .
, l .
Let T ( r )
=: { t ( x ) = w · x : (cid:107) w (cid:107) 2 ≤ r } be the class of linear function with Lipschitz semi-norm
less
than r
and we show that for each t ∈ T ( L/ 2) , there exists f ∈ F with (cid:107) f (cid:107) F ≤ L and y 0 ∈ { 1 , . .
, K } such that h ( x ) =
[ f ( x )] y 0 ∈ H L .
Let’s t ( x ) =
w 0 · x with (cid:107) w 0 (cid:107) 2 ≤ L/ 2 , we construct the network f ( x ) =
W l σ l ( x l ) + b l , x i =
σ i ( W i − 1 x
i − 1 + b i − 1 ) , i = 1 , . . .
, l, x 0 = x as follows,
• x 1 = σ 1 ( W 1 x ) =
( σ 1 ( w 0 · x ) , σ 1 ( − w 0 · x ) , 0 , . . . , 0) • x k = σ k
( W k x k − 1 )
= ( σ k ([ x k − 1 ] 1 ) , σ k ([ x k − 1 ] 2 ) , 0 , . .
k
= 2 , . . .
, l −• x l = W l x l − 1 =
([ x l − 1 ] 1 − [ x l − 1 ] 2 , 0 , . .
By construction above, we let h ( x ) =
[ f ( x )] 1 , h
( x ) =
σ 1 ( w 0 · x ) − σ 1 ( − w 0 · x ) , = w 0 · x, where (cid:107) f (cid:107) F ≤ Π li =1 (cid:107) W i (cid:107) σ = 2 L/ 2 = L , and thus h ∈ H L by deﬁnition.
Therefore,
[Equation]
where the second equality is implied from Cauchy-Schwarz inequality and the last inequality is implied from Khintchine inequality.
D. P ROOF OF T HEOREM
Proof of Theorem 1.
Consider l ( γ 1 ,γ 2 ) ( ζ ( ˜ f ( x ) , y )) , where ˜ f :
= f/L f is the normalized network, ζ ( ˜ f ( x ) , y ) ∈ G 1 .
Then for any γ 2 > γ 1 ≥ 0 ,
[Equation]
where the ﬁrst and last inequality is implied from 1[ ζ < γ 1 ] ≤ (cid:96) ( γ 1 ,γ 2 ) ( ζ ) ≤ 1[ ζ < γ 2 ] , the second inequality is a direct consequence of Lemma D.1, the third inequality results from Rademacher
Contraction Inequality (Lemma D.3) and ﬁnally the fourth equation is implied from Lemma D.4.
D. P ROOF OF T HEOREM
Proof of Theorem 2.
Firstly, we show after normalization, the normalize margin has an upper bound,
[Equation]
where x i = σ i ( W i x
i − 1 + b i ) with x 0 = x , ¯ W i =
( W i , b i ) and L σ i is the Lipschitz constant of activation function σ i with σ i (0)
= 0 , i = 1 , . . .
, L .
Then, for normalized network ˜ f
= f/L f with
L f = Π Li =
( L σ i (cid:107) ¯ W i (cid:107) σ ) and (cid:107) x (cid:107) 2 ≤ M , (cid:107)
[Equation]
Therefore ζ ( ˜ f ( x ) , y ) ≤ 2 (cid:107) ˜ f
( x ) (cid:107) 2 =
( M + L )
=:
M 1 , and the quantile margin is also bounded ˆ γ q,t ≤ M 1 for all q ∈ (0 , 1) , t = 1 , . .
, T .
The remaining proof is standard.
For any (cid:15) > 0 , we take a sequence of (cid:15) k and γ k , k
= 1 , 2 , . . .
by (cid:15) k
= (cid:15) + (cid:113) log kn and γ k = M 1 2 − k .
Then by Theorem 1,
[Equation]
where A k is the event P [ ζ ( ˜ f t ( x ) , y ) < 0] > P n [ ζ ( ˜ f ( x ) , y ) <
γ k ] + 2 K 2 γ k R
( H 1 )
+ (cid:15) k , and the probability is taken over samples { x 1 , ...
x n } .
We further consider the probability for none of A k occurs,
[Equation]
Hence, ﬁx a q ∈ [0 , 1] , for any t = 1 , . .
, T , as long as ˆ γ q,t > 0 , there exists a ˆ k ≥ 1 such that,
[Equation]
Therefore,
[Equation]
⊇
[Equation]
y
[Equation] [Equation]
P
[Equation]
The ﬁrst inequality is implied from P n [ ζ ( ˜ f t ( x ) , y ) <
ˆ γ q,t ] > P n [ ζ ( ˜ f t ( x ) , y ) <
γ ˆ k +1 ] , since γ ˆ k +1 ≤ ˆ γ q,t .
The second inequality is implied from ˆ γ q,t < 2 γ ˆ k +
and thus, 1 /γ ˆ k
+1 < 2 / ˆ γ q,t .
The third equality is the direct deﬁnition of (cid:15) ˆ k .
The last inequality is implied from ˆ k + 1 = log 2 ( M 1 /γ ˆ k +1 ) and again, 1 /γ ˆ k
+
<
2 / ˆ γ q,t .
The conclusion is proved immediately if we do a transform from (cid:15) to δ .