F IXUP I NITIALIZATION : R ESIDUAL L EARNING W ITHOUT N ORMALIZATION
yann@dauphin io
Tengyu Ma tengyuma@
A BSTRACT
Normalization layers are a staple in state-of-the-art deep neural network architectures.
They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic.
In this work, we challenge the commonly-held beliefs by showing that none of the perceived beneﬁts is unique to normalization.
Speciﬁcally, we propose ﬁxed-update initialization (Fixup), an initialization motivated by solving the exploding and vanishing gradient problem at the beginning of training via properly rescaling a standard initialization.
We ﬁnd training residual networks with Fixup to be as stable as training with normalization — even for networks with 10,000 layers.
Furthermore, with proper regularization, Fixup enables residual networks without normalization to achieve state-of-the-art performance in image classiﬁcation and machine translation.
I NTRODUCTION
Artiﬁcial intelligence applications have witnessed major advances in recent years.
At the core of this revolution is the development of novel neural network models and their training techniques.
For example, since the landmark work of He et al.
, most of the state-of-the-art image recognition systems are built upon a deep stack of network blocks consisting of convolutional layers and additive skip connections, with some normalization mechanism (e.g., batch normalization (Ioffe & Szegedy, 2015)) to facilitate training and generalization.
Besides image classiﬁcation, various normalization techniques (Ulyanov et al., 2016; Ba et al., 2016; Salimans & Kingma, 2016; Wu & He, 2018) have been found essential to achieving good performance on other tasks, such as machine translation (Vaswani et al., 2017) and generative modeling (Zhu et al., 2017).
They are widely believed to have multiple beneﬁts for training very deep neural networks, including stabilizing learning, enabling higher learning rate, accelerating convergence, and improving generalization.
Despite the enormous empirical success of training deep networks with normalization, and recent progress on understanding the working of batch normalization (Santurkar et al., 2018), there is currently no general consensus on why these normalization techniques help training residual neural networks.
Intrigued by this topic, in this work we study
(i) without normalization, can a deep residual network be trained reliably?
(And if so,) (ii) without normalization, can a deep residual network be trained with the same learning rate, converge at the same speed, and generalize equally well (or even better)?
Perhaps surprisingly, we ﬁnd the answers to both questions are Yes .
In particular, we show:
• Why normalization helps training.
We derive a lower bound for the gradient norm of a residual network at initialization, which explains why with standard initializations, normalization techniques are essential for training deep residual networks at maximal learning rate.
(Section 2) ∗ Work done at Facebook.
Equal contribution.
† Work done at Facebook.
Equal contribution.
‡ Work done at Facebook.
• Training without normalization.
We propose Fixup, a method that rescales the standard initialization of residual branches by adjusting for the network architecture.
Fixup enables training very deep residual networks stably at maximal learning rate without normalization.
(Section 3) • Image classiﬁcation.
We apply Fixup to replace batch normalization on image classiﬁcation benchmarks CIFAR-10 (with Wide-ResNet) and ImageNet (with ResNet), and ﬁnd Fixup with proper regularization matches the well-tuned baseline trained with normalization.
(Section 4.2) • Machine translation.
We apply Fixup to replace layer normalization on machine translation benchmarks IWSLT and WMT using the Transformer model, and ﬁnd it outperforms the baseline and achieves new state-of-the-art results on the same architecture.
(Section 4.3)
[Figure]
In the remaining of this paper
[Figure]
we ﬁrst analyze the exploding gradient problem of residual networks at initialization in Section 2.
To solve this problem, we develop Fixup in Section 3.
In Section 4 we quantify the properties of Fixup and compare it against state-of-the-art normalization methods on real world benchmarks.
A comparison with related work is presented in Section 5.
P ROBLEM : R ES N ET WITH S TANDARD I NITIALIZATIONS L EAD TO E XPLODING G RADIENTS
Standard initialization methods (Glorot & Bengio, 2010; He et al., 2015; Xiao et al., 2018) attempt to set the initial parameters of the network such that the activations neither vanish nor explode.
Unfortunately, it has been observed that without normalization techniques such as BatchNorm they do not account properly for the effect of residual connections and this causes exploding gradients.
Balduzzi et al.
(2017) characterizes this problem for ReLU networks, and we will generalize this to residual networks with positively homogenous activation functions.
A plain (i.e. without normalization layers) ResNet with residual blocks { F 1 , . .
. , F L } and input x 0 computes the activations as
[Equation]
ResNet output variance grows exponentially with depth.
Here we only consider the initialization, view the input x 0 as ﬁxed, and consider the randomness of the weight initialization.
We analyze the variance of each layer x l , denoted by Var[ x l ] (which is technically deﬁned as the sum of the variance of all the coordinates of x l .)
For simplicity we assume the blocks are initialized to be zero mean, i.e., E [ F l ( x l ) | x l ] = 0 .
By x l +1 = x l + F l ( x l ) , and the law of total variance, we have
Var[ x l +1 ] =
E [Var[ F ( x l ) | x l ]] + Var( x l ) .
Resnet structure prevents x l from vanishing by forcing the variance to grow with depth, i.e. Var[ x l ] <
Var[ x l +1 ] if E [Var[ F ( x l ) | x l ]] > 0 .
Yet, combined with initialization methods such as He et al.
, the output variance of each residual branch
Var[ F l ( x l ) | x l ] will be about the same as its input variance Var[ x l ] , and thus Var[ x l +1 ] ≈ 2Var
[ x l ] .
This causes the output variance to explode exponentially with depth without normalization (Hanin &
Rolnick, 2018) for positively homogeneous blocks (see Deﬁnition 1).
This is detrimental to learning because it can in turn cause gradient explosion.
As we will show, at initialization, the gradient norm of certain activations and weight tensors is lower bounded by the cross-entropy loss up to some constant.
Intuitively, this implies that blowup in the logits will cause gradient explosion.
Our result applies to convolutional and linear weights in a neural network with ReLU nonlinearity (e.g., feed-forward network, CNN), possibly with skip connections (e.g., ResNet, DenseNet), but without any normalization.
Our analysis utilizes properties of positively homogeneous functions, which we now introduce.
Deﬁnition 1 (positively homogeneous function of ﬁrst degree) .
A function f : R m → R n is called positively homogeneous (of ﬁrst degree)
(p.h.)
if for any input x ∈ R m and α > 0 , f ( α x ) =
αf ( x ) .
Deﬁnition 2 (positively homogeneous set of ﬁrst degree) .
Let θ =
{ θ i } i ∈ S be the set of parameters of f ( x ) and θ ph = { θ i }
i ∈ S ph ⊂ S .
We call θ ph a positively homogeneous set (of ﬁrst degree) (p.h. set) if for any α > 0 , f ( x ; θ \ θ ph , αθ ph ) = αf ( x ; θ \ θ ph , θ ph ) , where αθ ph denotes { αθ i } i ∈ S
ph .
Intuitively, a p.h. set is a set of parameters θ ph in function f such that for any ﬁxed input x and ﬁxed parameters θ \ θ ph , ¯ f ( θ ph ) (cid:44) f
( x ; θ \ θ ph , θ ph ) is a p.h. function.
Examples of p.h. functions are ubiquitous in neural networks, including various kinds of linear operations without bias (fully-connected (FC) and convolution layers, pooling, addition, concatenation and dropout etc.)
as well as ReLU nonlinearity.
Moreover, we have the following claim:
Proposition 1.
A function that is the composition of p.h. functions is itself p.h.
We study classiﬁcation problems with c classes and the cross-entropy loss.
We use f to denote a neural network function except for the softmax layer.
Cross-entropy loss is deﬁned as (cid:96) ( z , y ) (cid:44) − y T
( z − logsumexp ( z )) where y is the one-hot label vector, z (cid:44) f
( x ) ∈ R c is the logits where z i denotes its i -th element, and logsumexp ( z ) (cid:44) log (cid:16)(cid:80) i ∈
[ c ] exp( z i ) (cid:17) .
Consider a minibatch of training examples D M = { ( x ( m ) , y ( m ) )
} Mm =1 and the average cross-entropy loss (cid:96) avg ( D M ) (cid:44)
M (cid:80) Mm =1 (cid:96) ( f ( x ( m ) ) , y ( m ) ) , where we use ( m ) to index quantities referring to the m -th example.
(cid:107) · (cid:107) denotes any valid norm.
We only make the following assumptions about the network f :
f is a sequential composition of network blocks { f i } Li =1 , i.e. f ( x 0 ) =
f L ( f L − 1 ( . . .
f 1 ( x 0 ))) , each of which is composed of p.h. functions.
Weight elements in the FC layer are i.i
d. sampled from a zero-mean symmetric distribution.
These assumptions hold at initialization if we remove all
the
normalization layers
in
a
residual
network
with
ReLU nonlinearity,
assuming
all
the
biases
are initialized at 0 .
Our results are summarized in the following two theorems, whose proofs are listed in the appendix:
Theorem 1.
Denote the input to the i -th block by x i − 1 .
With Assumption 1, we have
[Equation]
where p is the softmax probabilities and H denotes the Shannon entropy.
Since H ( p ) is upper bounded by log( c ) and (cid:107) x
i − 1 (cid:107) is small in the lower blocks, blowup in the loss will cause large gradient norm with respect to the lower block input.
Our second theorem proves a lower bound on the gradient norm of a p.h. set in a network.
Theorem 2.
With Assumption 1, we have
[Equation]
Furthermore, with Assumptions 1 and 2, we have
[Equation]
It remains to identify such p.h. sets in a neural network.
In Figure 2 we provide three examples of p.h. sets in a ResNet without normalization.
Theorem 2 suggests that these layers would suffer from the exploding gradient problem, if the logits z blow up at initialization, which unfortunately would occur in a ResNet without normalization if initialized in a traditional way.
This motivates us to introduce a new initialization in the next section.
[Figure]
F IXUP : U PDATE A R ESIDUAL N ETWORK Θ( η ) PER SGD S TEP
Our analysis in the previous section points out the failure mode of standard initializations for training deep residual network: the gradient norm of certain layers is in expectation lower bounded by a quantity that increases indeﬁnitely with the network depth.
However, escaping this failure mode does not necessarily lead us to successful training — after all, it is the whole network as a function that we care about, rather than a layer or a network block.
In this section, we propose a top-down design of a new initialization that ensures proper update scale to the network function, by simply rescaling a standard initialization.
To start, we denote the learning rate by η and set our goal: f ( x ; θ ) is updated by Θ( η ) per SGD step after initialization as η → 0 .
That is, (cid:107) ∆ f ( x ) (cid:107) =
Θ( η ) where ∆ f ( x ) (cid:44) f
( x ; θ − η ∂∂θ (cid:96) ( f ( x ) , y ))
− f ( x ; θ ) .
Put another way, our goal is to design an initialization such that SGD updates to the network function are in the right scale and independent of the depth.
We deﬁne the Shortcut as the shortest path from input to output in a residual network.
The Shortcut is typically a shallow network with a few trainable layers.
1 We assume the Shortcut is initialized using a standard method, and focus on the initialization of the residual branches.
Residual branches update the network in sync.
To start, we ﬁrst make an important observation that the SGD update to each residual branch changes the network output in highly correlated directions.
This implies that if a residual network has L residual branches, then an SGD step to each residual branch should change the network output by Θ( η/L ) on average to achieve an overall Θ( η ) update.
We defer the formal statement and its proof until Appendix B.1.
Study of a scalar branch.
Next we study how to initialize a residual branch with m layers so that its SGD update changes the network output by Θ( η/L ) .
We assume m is a small positive integer (e.g., 2 or 3).
As we are only concerned about the scale of the update, it is sufﬁciently instructive to study the scalar case, i.e., F ( x ) =
( (cid:81) mi =1 a i ) x where a 1 , . .
. , a m , x ∈ R
+ .
For example, the standard initialization methods typically initialize each layer so that the output (after nonlinear activation) preserves the input variance, which can be modeled as setting ∀ i ∈ [ m ] , a i = 1 .
In turn, setting a i to a positive number other than 1 corresponds to rescaling the i-th layer by a i .
Through deriving the constraints for F ( x ) to make Θ( η/L ) updates, we will also discover how to rescale the weight layers of a standard initialization as desired.
In particular, we show the SGD update to F ( x ) is Θ( η/L ) if and only if the initialization satisﬁes the following constraint:
[Equation]
where j ∈ arg min k a k
[Equation] [Equation]
We defer the derivation until Appendix B.2.
Equation (5) suggests new methods to initialize a residual branch through rescaling the standard initialization of i-th layer in a residual branch by its corresponding scalar a i .
For example, we could set ∀ i ∈ [ m ] , a i =
L − 1 2 m − 2 .
Alternatively, we could start the residual branch as a zero function by setting a m = 0 and ∀ i ∈ [ m − 1] , a i =
L − 1 2 m − 2 .
In the second option, the residual branch does not need to “unlearn” its potentially bad random initial state, which can be beneﬁcial for learning.
Therefore, we use the latter option in our experiments, unless otherwise speciﬁed.
The effects of biases and multipliers.
With proper rescaling of the weights in all the residual branches, a residual network is supposed to be updated by Θ( η ) per SGD step — our goal is achieved.
However, in order to match the training performance of a corresponding network with normalization, there are two more things to consider: biases and multipliers.
Using biases in the linear and convolution layers is a common practice.
In normalization methods, bias and scale parameters are typically used to restore the representation power after normalization.
Intuitively, because the preferred input/output mean of a weight layer may be different from the preferred output/input mean of an activation layer, it also helps to insert bias terms in a residual network without normalization.
Empirically, we ﬁnd that inserting just one scalar bias before each weight layer and nonlinear activation layer signiﬁcantly improves the training performance.
Multipliers scale the output of a residual branch, similar to the scale parameters in batch normalization.
They have an interesting effect on the learning dynamics of weight layers in the same branch.
Speciﬁcally, as the stochastic gradient of a layer is typically almost orthogonal to its weight, learning rate decay tends to cause the weight norm equilibrium to shrink when combined with L2 weight decay (van Laarhoven, 2017).
In a branch with multipliers, this in turn causes the growth of the multipliers, increasing the effective learning rate of other layers.
In particular, we observe that inserting just one scalar multiplier per residual branch mimics the weight norm dynamics of a network with normalization, and spares us the search of a new learning rate schedule.
Put together, we propose the following method to train residual networks without normalization:
Fixup initialization (or: How to train a deep residual network without normalization) 1.
Initialize the classiﬁcation layer and the last layer of each residual branch to 0 .
Initialize every other layer using a standard method (e
g., He et al.
, and scale only the weight layers inside residual branches by L − 1 2 m − 2 . 3.
Add a scalar multiplier (initialized at 1) in every branch and a scalar bias (initialized at 0) before each convolution, linear, and element-wise activation layer.
It is important to note that Rule 2 of Fixup is the essential part as predicted by Equation (5).
Indeed, we observe that using Rule 2 alone is sufﬁcient and necessary for training extremely deep residual networks.
On the other hand, Rule 1 and Rule 3 make further improvements for training so as to match the performance of a residual network with normalization layers, as we explain in the above text.
3 We ﬁnd ablation experiments conﬁrm our claims (see Appendix C.1).
Our initialization and network design is consistent with recent theoretical work Hardt & Ma (2016);
Li et al.
(2018), which, in much more simpliﬁed settings such as linearized residual nets and quadratic neural nets, propose that small initialization tend to stabilize optimization and help generalizaiton.
However, our approach suggests that more delicate control of the scale of the initialization is beneﬁcial.
E XPERIMENTS 4. T RAINING AT INCREASING DEPTH
One of the key advatanges of BatchNorm is that it leads to fast training even for very deep models (Ioffe & Szegedy, 2015).
Here we will determine if we can match this desirable property by relying only on proper initialization.
We propose to evaluate how each method affects training very deep nets by measuring the test accuracy after the ﬁrst epoch as we increase depth .
In particular, we use the wide residual network (WRN) architecture with width 1 and the default weight decay 5e −(Zagoruyko & Komodakis, 2016).
We speciﬁcally use the default learning rate of 0 . 1 because the ability to use high learning rates is considered to be important to the success of BatchNorm.
We compare Fixup against three baseline methods — (1) rescale the output of each residual block by 1 √(Balduzzi et al., 2017), (2) post-process an orthogonal initialization such that the output variance of each residual block is close to 1 (Layer-sequential unit-variance orthogonal initialization, or LSUV)
(Mishkin & Matas, 2015), (3) batch normalization (Ioffe & Szegedy, 2015).
We use the default batch size of 128 up to 1000 layers, with a batch size of 64 for 10,000 layers.
We limit our budget of epochs to 1 due to the computational strain of evaluating models with up to 10,000 layers.
[Figure]
Figure 3 shows the test accuracy at the ﬁrst epoch as depth increases.
Observe that Fixup matches the performance of BatchNorm at the ﬁrst epoch, even with 10,000 layers.
LSUV and (cid:112) 1 / 2 -scaling are not able to train with the same learning rate as BatchNorm past 100 layers.
4. I MAGE CLASSIFICATION
In this section, we evaluate the ability of Fixup to replace batch normalization in image classiﬁcation applications.
On the CIFAR-10 dataset, we ﬁrst test on ResNet-110 (He et al., 2016) with default hyper-parameters; results are shown in Table 1.
Fixup obtains 7% relative improvement in test error compared with standard initialization; however, we note a substantial difference in the difﬁculty of training.
While network with Fixup is trained with the same learning rate and converge as fast as network with batch normalization, we fail to train a Xavier initialized ResNet-110 with 0.1x maximal learning rate.
The test error gap in Table 1 is likely due to the regularization effect of BatchNorm rather than difﬁculty in optimization; when we train Fixup networks with better regularization, the test error gap disappears and we obtain state-of-the-art results on CIFAR-10 and SVHN without normalization layers (see Appendix C.2).
[Table]
On the ImageNet dataset, we benchmark Fixup with the ResNet-50 and ResNet-101 architectures (He et al., 2016), trained for 100 epochs and 200 epochs respectively.
Similar to our ﬁnding on the CIFAR-10 dataset, we observe that (1) training with Fixup is fast and stable with the default hyperparameters, (2) Fixup alone signiﬁcantly improves the test error of standard initialization, and (3) there is a large test error gap between Fixup and BatchNorm.
Further inspection reveals that
Fixup initialized models obtain signiﬁcantly lower training error compared with BatchNorm models (see Appendix C.3), i.e., Fixup suffers from overﬁtting.
We therefore apply stronger regularization to the Fixup models using Mixup (Zhang et al., 2017).
We ﬁnd it is beneﬁcial to reduce the learning rate of the scalar multiplier and bias by 10 x when additional large regularization is used.
Best
Mixup coefﬁcients are found through cross-validation: they are 0 . 2 , 0 . 1 and 0 .
7 for BatchNorm,
GroupNorm (Wu & He, 2018) and Fixup respectively.
We present the results in Table 2, noting that with better regularization, the performance of Fixup is on par with GroupNorm.
[Table]
4. M ACHINE TRANSLATION
To demonstrate the generality of Fixup
[Table]
we also apply it to replace layer normalization (Ba et al., 2016) in Transformer (Vaswani et al., 2017)
[Table]
a state-of-the-art neural network for machine translation.
Speciﬁcally, we use the fairseq library (Gehring et al., 2017) and follow the Fixup template in Section 3 to modify the baseline model.
We evaluate on two standard machine translation datasets, IWSLT German-English (de-en) and WMT English-German (en-de) following the setup of Ott et al.
For the IWSLT de-en dataset
[Table]
we cross-validate the dropout probability from { 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 } and ﬁnd 0 .
5 to be optimal for both Fixup and the LayerNorm baseline.
For the
WMT’16 en-de dataset, we use dropout probability 0 .
All models are trained for 200 k updates.
It was reported (Chen et al., 2018) that “Layer normalization is most critical to stabilize the training process... removing layer normalization results in unstable training runs”.
However we ﬁnd training with Fixup to be very stable and as fast as the baseline model.
Results are shown in Table 3.
Surprisingly, we ﬁnd the models do not suffer from overﬁtting when LayerNorm is replaced by
Fixup, thanks to the strong regularization effect of dropout.
Instead, Fixup matches or supersedes the state-of-the-art results using Transformer model on both datasets.
[Table]
R ELATED W ORK
Normalization methods.
Normalization methods have enabled training very deep residual networks, and are currently an essential building block of the most successful deep learning architectures.
All normalization methods for training neural networks explicitly normalize (i.e. standardize) some component (activations or weights) through dividing activations or weights by some real number computed from its statistics and/or subtracting some real number activation statistics (typically the mean) from the activations.
In contrast, Fixup does not compute statistics (mean, variance or norm) at initialization or during any phase of training, hence is not a normalization method.
Theoretical analysis of deep networks.
Training very deep neural networks is an important theoretical problem.
Early works study the propagation of variance in the forward and backward pass for different activation functions (Glorot & Bengio, 2010; He et al., 2015).
Recently, the study of dynamical isometry (Saxe et al., 2013) provides a more detailed characterization of the forward and backward signal propogation at initialization (Pennington et al., 2017; Hanin, 2018), enabling training 10,000-layer CNNs from scratch (Xiao et al., 2018).
For residual networks, activation scale (Hanin & Rolnick, 2018), gradient variance (Balduzzi et al., 2017) and dynamical isometry property (Yang & Schoenholz, 2017) have been studied.
Our analysis in Section 2 leads to the similar conclusion as previous work that the standard initialization for residual networks is problematic.
However, our use of positive homogeneity for lower bounding the gradient norm of a neural network is novel, and applies to a broad class of neural network architectures (e.g., ResNet,
DenseNet) and initialization methods (e.g., Xavier, LSUV) with simple assumptions and proof.
Hardt & Ma (2016) analyze the optimization landscape (loss surface) of linearized residual nets in the neighborhood around the zero initialization where all the critical points are proved to be global minima.
Yang & Schoenholz (2017) study the effect of the initialization of residual nets to the test performance and pointed out Xavier or He initialization scheme is not optimal.
In this paper, we give a concrete recipe for the initialization scheme with which we can train deep residual networks without batch normalization successfully.
Understanding batch normalization.
Despite its popularity in practice, batch normalization has not been well understood.
Ioffe & Szegedy (2015) attributed its success to “reducing internal covariate shift”, whereas Santurkar et al.
(2018) argued that its effect may be “smoothing loss surface”.
Our analysis in Section 2 corroborates the latter idea of Santurkar et al.
by showing that standard initialization leads to very steep loss surface at initialization.
Moreover, we empirically showed in Section 3 that steep loss surface may be alleviated for residual networks by using smaller initialization than the standard ones such as Xavier or He’s initialization in residual branches.
van
Laarhoven (2017); Hoffer et al.
(2018) studied the effect of (batch) normalization and weight decay on the effective learning rate.
Their results inspire us to include a multiplier in each residual branch.
ResNet initialization in practice.
Gehring et al.
Balduzzi et al.
(2017) proposed to address the initialization problem of residual nets by using the recurrence x l = (cid:112) 1 / 2 ( x l − 1 + F l ( x l − 1 )) .
Mishkin & Matas (2015) proposed a data-dependent initialization to mimic the effect of batch normalization in the ﬁrst forward pass.
While both methods limit the scale of activation and gradient, they would fail to train stably at the maximal learning rate for very deep residual networks, since they fail to consider the accumulation of highly correlated updates contributed by different residual branches to the network function (Appendix B.1).
Srivastava et al.
; Hardt & Ma (2016);
Goyal et al.
; Kingma & Dhariwal (2018) found that initializing the residual branches at (or close to) zero helped optimization.
Our results support their observation in general, but Equation (5) suggests additional subtleties when choosing a good initialization scheme.
C ONCLUSION
In this work, we study how to train a deep residual network reliably without normalization.
Our theory in Section 2 suggests that the exploding gradient problem at initialization in a positively homogeneous network such as ResNet is directly linked to the blowup of logits.
In Section 3 we develop Fixup initialization to ensure the whole network as well as each residual branch gets updates of proper scale, based on a top-down analysis.
Extensive experiments on real world datasets demonstrate that Fixup matches normalization techniques in training deep residual networks, and achieves state-of-the-art test performance with proper regularization.
Our work opens up new possibilities for both theory and applications.
Can we analyze the training dynamics of Fixup, which may potentially be simpler than analyzing models with batch normalization is?
Could we apply or extend the initialization scheme to other applications of deep learning?
It would also be very interesting to understand the regularization beneﬁts of various normalization methods, and to develop better regularizers to further improve the test performance of Fixup.
A CKNOWLEDGMENTS
The authors would like to thank Yuxin Wu, Kaiming He, Aleksander Madry and the anonymous reviewers for their helpful feedback.
A P ROOFS FOR S ECTIONA. G RADIENT NORM LOWER BOUND FOR THE INPUT TO A NETWORK BLOCK
Proof of Theorem 1.
We use f i → j to denote the composition f
j ◦ f j − 1 ◦ · · · ◦ f i , so that z = f i
→ L ( x i − 1 ) for all
i ∈
[ L ] .
Note that z is p.h. with respect to the input of each network block, i.e. f i → L ((1 + (cid:15) )
x i − 1 ) =
(1 + (cid:15) )
f
i →
L ( x i − 1 ) for (cid:15) > − 1 .
This allows us to compute the gradient of the cross-entropy loss with respect to the scaling factor (cid:15) at (cid:15)
=
as
[Equation]
Since the gradient L 2 norm (cid:107) ∂(cid:96) /
∂ x i − 1 (cid:107) must be greater than the directional derivative ∂∂t (cid:96) ( f i →
L ( x i − 1 + t x
i − 1 (cid:107) x
i − 1 (cid:107) ) , y ) , deﬁning (cid:15) =
t / (cid:107) x
i − 1 (cid:107) we have (cid:13)(cid:13)(cid:13)(cid:13) ∂(cid:96)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13) ≥
[Equation]
(cid:96)
[Equation] [Equation]
A. G RADIENT NORM LOWER BOUND FOR POSITIVELY HOMOGENEOUS SETS
Proof of Theorem 2.
The proof idea is similar.
Recall that if θ ph is a p.h. set, then ¯ f ( m ) ( θ ph ) (cid:44) f
( x ( m ) ; θ \ θ ph , θ ph ) is a p.h. function.
We therefore have
[Equation]
hence we again invoke the directional derivative argument to show
[Equation]
In order to estimate the scale of this lower bound, recall the FC layer weights are i
[Equation]
i
[Equation]
d. sampled from a symmetric, mean-zero distribution, therefore z has a symmetric probability density function with mean 0 .
We hence have
[Equation]
where the inequality uses the fact that logsumexp ( z ) ≥ max i ∈ [
[Equation]
] z
[Equation]
the last equality is due to y and z being independent at initialization and E z = 0 .
Using the trivial bound E H ( p ) ≤ log( c ) , we get
[Equation]
which shows that the gradient norm of a p.h. set is of the order
Ω( E [max i ∈
[ c ] z i ]) at initialization.
B P ROOFS FOR S ECTIONB. R ESIDUAL BRANCHES UPDATE THE NETWORK IN SYNC
A common theme in previous analysis of residual networks is the scale of activation and gradient (Balduzzi et al., 2017; Yang & Schoenholz, 2017; Hanin & Rolnick, 2018).
However, it is more important to consider the scale of actual change to the network function made by a (stochastic) gradient descent step.
If the updates to different layers cancel out each other, the network would be stable as a whole despite drastic changes in different layers; if, on the other hand, the updates to different layers align with each other, the whole network may incur a drastic change in one step, even if each layer only changes a tiny amount.
We now provide analysis showing that the latter scenario more accurately describes what happens in reality at initialization.
For our result in this section, we make the following assumptions:
• f is a sequential composition of network blocks { f i } Li =1 , i.e. f ( x 0 ) =
f L ( f L − 1 ( . . .
f 1 ( x 0 ))) , consisting of fully-connected weight layers, ReLU activation functions and residual branches.
• f L is a fully-connected layer with weights i.i.d.
sampled from a zero-mean distribution.
• There is no bias parameter in f .
For l < L , let x l − 1 be the input to f l and F l ( x l − 1 ) be a branch in f l with m l layers.
Without loss of generality, we study the following speciﬁc form of network architecture:
[Equation]
For the last block we denote m L = 1 and f L ( x L − 1 ) =
F L ( x L − 1 ) =
W (1)
L x L − 1 .
Furthermore, we always choose 0 as the gradient of ReLU when its input is 0 .
As such, with input x , the output and gradient of ReLU ( x ) can be simply written as D 1 [ x > 0] x , where D 1 [ x > 0] is a diagonal matrix with diagonal entries corresponding to 1 [ x > 0] .
Denote the preactivation of the i -th layer
(i.e. the input to the i -th ReLU) in the l -th block by x ( i ) l .
We deﬁne the following terms to simplify our presentation:
[Equation]
We have the following result on the gradient update to f :
Theorem 3.
With the above assumptions, suppose we update the network parameters by ∆ θ = − η ∂∂θ (cid:96) ( f ( x 0 ; θ ) , y ) , then the update to network output ∆ f ( x 0 ) (cid:44) f
( x 0 ; θ + ∆ θ ) − f ( x 0 ; θ
[Equation]
12) where z (cid:44) f
( x
) ∈ R c
is the
logits
Let us discuss the implecation of this result before delving into the proof.
As each J il is a c × c real symmetric positive semi-deﬁnite matrix, the trace norm of each J il equals its trace.
Similarly, the trace norm of J (cid:44) (cid:80) l (cid:80) i J il equals the trace of the sum of all J il as well, which scales linearly with the number of residual branches L .
Since the output z has no (or little) correlation with the target y at the start of training, ∂(cid:96)∂
z is a vector of some random direction.
It then follows that the expected update scale is proportional to the trace norm of J , which is proportional to L as well as the average trace of J il .
Simply put, to allow the whole network be updated by Θ( η ) per step independent of depth, we need to ensure each residual branch contributes only a Θ( η/L ) update on average.
Proof.
The ﬁrst insight to prove our result is to note that conditioning on a speciﬁc input x 0 , we can replace each ReLU activation layer by a diagonal matrix and does not change the forward and backward pass.
(In fact, this is valid even after we apply a gradient descent update, as long as the learning rate η > 0 is sufﬁciently small so that all positive preactivation remains positive.
This observation will be essential for our later analysis.)
We thus have the gradient w.r.t.
the i -th weight layer in the l -th block is
[Equation]
where ⊗ denotes the Kronecker product.
The second insight is to note that with our assumptions, a network block and its gradient w.r.t.
its input have the following relation:
[Equation] [Equation]
We then plug in Equation (13) to the gradient update ∆ θ = − η ∂∂θ (cid:96) ( f ( x 0 ; θ ) , y ) , and recalculate the forward pass f ( x 0 ; θ +∆ θ ) .
The theorem follows by applying Equation (14) and a ﬁrst-order Taylor series expansion in a small neighborhood of η = 0 where f ( x 0 ; θ + ∆ θ ) is smooth w.r.t.
η .
B. W HAT SCALAR BRANCH HAS Θ( η/L ) UPDATES ?
For this section, we focus on the proper initialization of a scalar branch F ( x ) =
( (cid:81) mi
=1 a i ) x .
We have the following result:
Theorem 4.
Assuming ∀ i, a i ≥ 0 , x = Θ(1) and
[Equation]
x ) = Θ(1) , then ∆ F ( x ) (cid:44) F ( x ; θ − η ∂(cid:96)∂θ ) −
F ( x ; θ ) is Θ( η/L )
if and only if
[Equation]
, where j ∈ arg min k a k
[Equation]
Proof.
We start by calculating the gradient of each parameter:
[Equation]
and a ﬁrst-order approximation of ∆ F ( x ) :
[Equation]
where we conveniently abuse some notations by deﬁning
[Equation]
Denote (cid:80) mi =1 1 a 2 i as M and min k a k as A , we have
[Equation]
19) and therefore by rearranging Equation (17) and letting ∆ F ( x ) =
Θ( η/L ) we get
[Equation]
i.e. F ( x ) /A
= Θ(1 / √ L ) .
Hence the “only if” part is proved.
For the “if” part, we apply Equation (19) to Equation (17) and observe that by Equation (15)
[Equation]
The result of this theorem provides useful guidance on how to rescale the standard initialization to achieve the desired update scale for the network function.
C A DDITIONAL EXPERIMENTS C. A BLATION STUDIES OF F IXUP
In this section we present the training curves of different architecture designs and initialization schemes.
Speciﬁcally, we compare the training accuracy of batch normalization, Fixup, as well as a few ablated options: (1) removing the bias parameters in the network; (2) use 0 .
1 x the suggested initialization scale and no bias parameters; (3) use 10 x the suggested initialization scale and no bias parameters; and (4) remove all the residual branches.
The results are shown in Figure 4.
We see that initializing the residual branch layers at a smaller scale (or all zero) slows down learning, whereas training fails when initializing them at a larger scale; we also see the clear beneﬁt of adding bias parameters in the network.
C. CIFAR AND SVHN WITH BETTER REGULARIZATION
We perform additional experiments to validate our hypothesis that the gap in test error between
Fixup and batch normalization is primarily due to overﬁtting.
To combat overﬁtting, we use Mixup (Zhang et al., 2017) and Cutout (DeVries & Taylor, 2017) with default hyperparameters as additional regularization.
On the CIFAR-10 dataset, we perform experiments with WideResNet-40and on SVHN we use WideResNet-16-12 (Zagoruyko & Komodakis, 2016), all with the default hyperparameters.
We observe in Table 4 that models trained with Fixup and strong regularization are competitive with state-of-the-art methods on CIFAR-10 and SVHN, as well as our baseline with batch normalization.
Batch Index
T r a i n
A cc u r ac y ( % )
BatchNorm
Fixup
L 1 2 m 2 , no bias 0.1 L 1 2 m10 L 1 2 mno residual
Dataset
Model
Normalization
Test Error ( % )
CIFAR- (Zagoruyko & Komodakis, 2016)
Yes 3 .(Yamada et al., 2018) 2.
BatchNorm + Mixup + Cutout 2.
(Graham, 2014)
No 3.
Fixup-init + Mixup + Cutout 2.
SVHN (Zagoruyko & Komodakis, 2016)
Yes 1.
(DeVries & Taylor, 2017) 1.
BatchNorm + Mixup + Cutout 1.
(Lee et al., 2016)
No 1.
Fixup-init + Mixup + Cutout 1.
Figure 5 shows that without additional regularization Fixup ﬁts the training set very well, but overﬁts signiﬁcantly.
We see in Figure 6 that Fixup is competitive with networks trained with normalization when the Mixup regularizer is used.
[Figure]
We observe that Fixup is able to better ﬁt the training data and that leads to overﬁtting - more regularization is needed
[Figure]
D A DDITIONAL REFERENCES : A BRIEF HISTORY OF NORMALIZATION METHODS
The ﬁrst use of normalization in neural networks appears in the modeling of biological visual system and dates back at least to Heeger (1992) in neuroscience and to Pinto et al.
; Lyu & Simoncelli (2008) in computer vision, where each neuron output is divided by the sum (or norm) of all of the outputs, a module called divisive normalization.
Recent popular normalization methods, such as local response normalization (Krizhevsky et al., 2012), batch normalization (Ioffe & Szegedy, 2015) and layer normalization (Ba et al., 2016) mostly follow this tradition of dividing the neuron activations by their certain summary statistics, often also with the activation mean subtracted.
An exception is weight normalization (Salimans & Kingma, 2016), which instead divides the weight parameters by their statistics, speciﬁcally the weight norm; weight normalization also adopts the idea of activation normalization for weight initialization.
The recently proposed actnorm (Kingma & Dhariwal, 2018) removes the normalization of weight parameters, but still use activation normalization to initialize the afﬁne transformation layers.