P -BN: T OWARDS E FFECTIVE B ATCH N ORMALIZATION IN THE P ATH S PACE
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural networks with ReLU activation functions have demonstrated their success in many applications.
Recently, researchers noticed a potential issue with the optimization of ReLU networks: the ReLU activation functions are positively scaleinvariant (PSI), while the weights are not.
This mismatch may lead to undesirable behaviors in the optimization process.
Hence, some new algorithms that conduct optimizations directly in the path space (the path space is proven to be PSI) were developed, such as Stochastic Gradient Descent (SGD) in the path space, and it was shown that, SGD in the path space is superior to that in the weight space.
However, it is still unknown that whether other deep learning techniques beyond
SGD, such as batch normalization (BN), could also have their counterparts in the path space.
In this paper, we conduct a formal study on the design of BN in the path space.
According to our study, the key challenge is how to ensure the forward propagation in the path space, because BN is utilized during the forward process.
To tackle such challenge, we propose a novel re-parameterization of ReLU networks, with which we replace each weight in the original neural network, with a new value calculated from one or several paths, while keeping the outputs of the network unchanged for any input.
Then we show that BN in the path space, namely P -BN, is just a slightly modiﬁed conventional BN on the re-parameterized
ReLU networks.
Finally, our experiments on two benchmark datasets, CIFAR and
ImageNet, show that the proposed P -BN can signiﬁcantly outperform the conventional BN in the weight space.
I NTRODUCTION
In recent years, neural networks with rectiﬁed linear unit (ReLU) activation functions (abbrev.
ReLU networks) , have been successfully applied to many domains, such as image classiﬁcation (He et al., 2016), game playing (Mnih et al., 2015), and text processing (Kim, 2014).
Recently, it has been noticed that the feedforward neural networks with ReLU, namely ReLU networks, are positively scale-invariant (PSI) (Neyshabur et al., 2015a; 2016), which means when the incoming and outgoing weights of a hidden node, are respectively multiplied and divided by a positive constant, the outputs of ReLU networks will keep unchanged for any input.
However, the vector space, composed of weights, in which the conventional optimization algorithms, e.g., Stochastic Gradient Descent (SGD), optimize the neural networks, is not PSI, and such mismatch may lead to undesirable behaviors in the optimization process (Neyshabur et al., 2015a).
To tackle such challenge, some recent studies have shown that, one ReLU network can be optimized in a completely new PSI parameter space, i.e., the path space , instead of its original weight space (Meng et al., 2019).
Detailedly, while regarding ReLU networks as directed acyclic graphs (DAGs), we can calculate the outputs of ReLU networks by using path-values, i.e., the multiplication of weights along each input-output path in the DAG.
Here, path-values are invariant to the positive rescaling of weights, which exactly matches the PSI property of ReLU networks, and the optimization algorithms (e.g., SGD) in the path space (i.e., the vector space composed of path-values), are shown to be superior to those in the weight space (Neyshabur et al., 2015a; 2016;
Meng et al., 2019).
There are already some optimization algorithms developed in the path space, however, to the best of our knowledge, it is still unknown whether other deep learning techniques beyond SGD, can also have their counterparts in the path space.
For example, BN is one of the most widely used normal- ization approaches (Ioffe & Szegedy, 2015; Santurkar et al., 2018), and is crucial for facilitating the training process of neural networks.
With the help of BN, the training process of neural networks can be accelerated, by both decoupling the scale and direction of weight vectors (Arora et al., 2018) and smoothing the optimization landscape (Santurkar et al., 2018).
[Figure]
In this paper, we conduct a formal study on the design of BN in the path space.
First , it is known that,
BN is successfully utilized during the forward process in the weight space, but how to ensure the forward propagation still remains unclear in the path space, i.e., the way to calculate the outputs of hidden nodes via path-values layer by layer is unknown.
For example, in Fig. 1, we show a network and its outputs calculated by both weights and path-values.
Here, the outputs of this network can be simply calculated by using path-values, because a path starts from an input node and ends up with an output node, and the path-value is just the multiplication of weights along this path.
Nevertheless, the outputs of hidden nodes cannot be calculated by path-values in this obvious way.
Thus, to solve this problem, we propose a re-parameterization for the weights in ReLU networks, which can replace each weight in the original network, with a new value calculated by one or several path-values, and the outputs of the re-parameterized network will keep unchanged for any input.
Therefore, the forward propagation in the path space can be easily conducted after re-parameterizing the network.
Then , we show that BN in the path space is just a slightly modiﬁed conventional BN on the reparameterized network.
Considering that BN in the weight space, can make the gradient propagation more effective, we study BN in the path space from the perspective of effective propagation.
In details, we apply the conventional BN to the re-parameterized network, and ﬁnd that the radients will explode during the backward propagation in the path space.
Thus, we propose a novel BN in the path space, namely P -BN , which slightly modiﬁes the conventional BN, via excluding the term whose coefﬁcient is ﬁxed by re-parameterization from normalization.
Particularly, we theoretically prove and experimentally show that compared with the conventional BN, P -BN can propagate gradients in the path space more effectively.
Finally , we conduct extensive experiments on multiple datasets, including CIFAR, ImageNet, with multiple network structures, including ResNet and PlainNet.
The results demonstrate that the P -BN based optimization algorithms can signiﬁcantly outperform the ones with the conventional BN.
B ACKGROUND 2. R ELATED W ORK
In recent years, some researchers have conducted many theoretical studies on the path of ReLU networks.
On the one hand , in terms of optimization, for example, Neyshabur et al.
(2015a) proposed a new regularization term, i.e., path-norm, and the Path-SGD algorithm for minimizing the regularized loss.
Meng et al.
(2019) proposed G -SGD algorithm to directly optimize the model in the path space, by utilizing the gradients w.r.t.
path-values.
On the other hand , in terms of generalization, for example, Neyshabur et al.
(2015b)
and Zheng et al.
(2018) gave the relationship between generalization error and path-norm or basis path-norm, respectively.
Besides, E et al.
(2018) leveraged the path representation of ReLU networks, to analyze the generalization error of them.
However, the aforementioned studies have not involved normalization approaches (Ba et al., 2016;
Wu & He, 2018), such as BN (Ioffe & Szegedy, 2015), which are proved to be much crucial for training neural networks.
In particular, as for ReLU networks, BN is especially widely used in computer vision domain (He et al., 2016;
Li et al., 2016; Huang et al., 2017).
Furthermore, there are also many theoretical studies on BN (Santurkar et al., 2018; Arora et al., 2018; Kohler et al., 2019).
Accordingly, based on the studies on optimizing ReLU networks in the path space, and the crucial normalization technique, i.e., BN, in this paper, we will give a formal study on designing BN in the path space.
We will next brieﬂy introduce the optimization of ReLU networks without BN in the path space.
2. O PTIMIZING R E LU NETWORKS IN THE P ATH S PACE
𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂𝑂
Input
Output 𝑂𝑤𝑤𝑤𝑤𝑤𝑤
Hidden
In this subsection, we introduce the feedforward ReLU network and the optimization algorithm in the path space.
Firstly , as shown in Fig. 2, consider an L -layer feedforward ReLU network without BN 1 whose weight matrices are { w l ; l = 1 , · · · , L } , we use O lj , l = 0 , · · · , L to denote the j-th node in the l-th layer.
Based on these above, we further use w ljk to represent the weight on the edge which connects O l − 1 k and O lj .
As for the outputs of hidden nodes, suppose that given a d -dimensional input x = ( x 1 , · · · , x d ) , then the output of O lj and the hidden vector in the l-th layer, can be denoted as o l j ( x ) and
o l ( x ) , respectively.
Thus, the outputs can be propagated as o l ( x ) =
g ( w l o l − 1 ( x )) , where g ( · ) = max ( · , 0) is a
ReLU activation function.
Secondly , we will show the concept of a path in ReLU networks, and brieﬂy introduce the optimization algorithm in the path space.
In details, regarding the network structure as a DAG consisting of nodes and edges, a path can be deﬁned as: a list of nodes or edges, starting from an input node, successively passing by several hidden nodes along the edges, and ﬁnally ending up with an output node.
For example, in Fig. 2, path p starts from O 02 , passes O 12 and O 21 , and ends up with O 33 .
And then, the path-value of path p can be deﬁned as a multiplication of the weights along p , i.e., v p = w 222 w 212 w 331 .
Therefore, output o Lj ( x ) can be calculated by using path-values as o Lj ( x ) = (cid:80) dk
=1 (cid:80) p ∈ P k,j v p · a p · x k .
Here, P is a set which contains the paths staring from O 0 k and ending up with O Lj , and a p , the ReLU activation status of path p , can be calculated as a p =
(cid:81)
O lpjp ∈ p
I ( o l p j p ( x ) > 0) , where O l p j p ∈ p represents that O l p j p is passed by p .
As for the optimization algorithm, we introduce two typical ones, i.e., Path-SGD (Neyshabur et al., 2015a) and G -SGD (Meng et al., 2019).
Path-SGD is to add a new regularizer, i.e., path-norm, into the loss function, and then heuristically utilizes a coordinate-wised solution for minimizing the pathnorm regularized loss.
It is shown to be superior to SGD for MLP, but with huge computational costs when applied to Convolutional Neural Networks (CNNs), due to the complicate correlations among path-values.
Besides, Path-SGD only utilizes the PSI property to propose the regularizer, but cannot optimize the network in the path space.
Comparatively, G -SGD can solve the computation problem, by decoupling correlations between path-values.
Besides, G -SGD can optimize the network via updating path-values by using gradients w.r.t.
them instead of weights.
Thus, G -SGD can serve as the SGD in the path space, and the detailed on G -SGD are listed in Appendix C.
T O D ESIGN BN IN THE P ATH S PACE
In this section, we ﬁrst introduce a re-parameterization process, to ensure the forward propagation in the path space, and calculate the outputs of hidden nodes by using path-values.
Then, we will show that the conventional BN cannot effectively propagate the error signals in the path space, which calls us to design an effective BN in the path space.
3. R E - PARAMETERIZATION
In this subsection, we introduce a re-parameterization for the parameters of ReLU networks, which can replace each weight with a path-value or a ratio of path-values, and the outputs of the re-
Rescaling Operation 𝑤 213 𝑤 113 =
𝑣 213 𝑣 111 , 𝑤 123 𝑤 223 =
𝑣 123 𝑣𝑤 122 𝑤 113 𝑤 222 𝑤 223 =
𝑣 122 𝑣𝑤 212 𝑤 223 𝑤 112 𝑤 113 =𝑣 212 𝑣 111 , 𝑤 121 𝑤 112 𝑤 113 =
𝑣 121 , 𝑤 111 𝑤
𝑤
=
𝑣 111 , 𝑤 211 𝑤 222 𝑤
=
𝑣 211 , 𝑤 221 𝑤 222 𝑤 223 =𝑣𝑤𝑤𝑤𝑤𝒙 𝟏𝒊 𝒙 𝟐𝒊 𝑤 121 𝑤𝑤
[Figure]
𝑤 122 𝑤𝑤𝑤 123 𝑤
[Figure]
𝟏 (𝒙 𝒊 )
[Figure]
𝒊 )
[Figure]
𝑤 112 𝑤 113 𝑤 112 𝑤     𝑤 222 𝑤𝑤𝒙 𝟏𝒊 𝒙 𝟐𝒊 𝑤𝑶 𝟏 (𝒙 𝒊 ) 𝑶 𝟐 (𝒙 𝒊 ) 𝑤 112 𝑤𝑤 113 𝑤𝑤 223 𝑤 223 𝑤 213 𝑤 113 𝑤 123 𝑤𝑤 212 𝑤 223 𝑤 122 𝑤𝑤𝑤𝑤
𝑤 113 𝑤 222 𝑤𝑤 211 𝑤 222 𝑤 223 𝑤 221 𝑤 222 𝑤𝑤 222 𝑤 223 𝑤 222 𝑤𝒙 𝟏𝒊 𝒙 𝟐𝒊 1 𝑣 213 𝑣 111 𝑣 123 𝑣 221𝑶
(𝒙 𝒊 )
𝑶 𝟐 (𝒙 𝒊 ) 𝑤 212 𝑤 223 𝑤 112 𝑤𝑤 111 𝑤 112 𝑤 113 𝑤 121 𝑤 112 𝑤𝑣 111 𝑣𝒙 𝟏𝒊 𝒙 𝟐𝒊 𝑣 121 𝑣1 𝑣 212 𝑣 111 𝑣 122 𝑣
[Figure]
(𝒙 𝒊 ) 𝑶 𝟐 (𝒙 𝒊 ) 𝑣 [Figure] 𝑣
[Figure]
𝑣
[Figure]
parameterized network will keep unchanged for any input.
First of all, we present the process of re-parameterization in Theorem 3.1.
Theorem 3.1 Consider an L -layer ReLU network with weight matrices { w l ; l = 1 , · · · , L } , if the diagonal elements in matrix w l , l > 2 are positive, the hidden outputs of the ReLU network can be calculated by using path-values as follows:
[Equation]
Here, g ( · ) denotes the ReLU activation function, and v 1 kk is the path-value of the path which only contains the k-th diagonal elements in the weight matrices, i.e., v 1 kk = (cid:81)
Ll =1 w lkk , and v ljk , k (cid:54) =
j is the path-value of the path which contains one non-diagonal weight, i.e., v ljk =
( (cid:81)
s<
l w skk ) · w ljk · ( (cid:81) s>l w ljj ) .
Proof: Based on the PSI property, we will prove Theorem 3.1 by designing a re-parameterization process which consists several weight rescaling steps.
After these steps, the ReLU network can be equivalently re-parameterized, which means that the outputs will keep unchanged for any input after re-parameterization.
Here, we will next demonstrate the re-parameterization process by showing that it can be conducted for a 3-layer MLP, as well as generalized to a multi-layer setting.
As for a 3-layer MLP , the process is shown in Fig.
3, and described in the following two steps:
(1) The weights connected with nodes
O 2 j , j = 1 , 2 are rescaled, i.e., the incoming and outgoing weights of node
O 2 j are multiplied and divided by w 3 jj , respectively.
To be speciﬁc, w 311 , w 321 and w 211 , w 212 are divided and multiplied by w 311 , and similarly, w 322 , w 312 and w 222 , w 221 are also rescaled by using w
According to the deﬁnition of a path value, we have w 321 w 311 =
w
w 211 w 111 w 311 w 211 w 111 = v 321 v 111 , w 312 w 322 =
w 312 w 222 w 122 w 322 w 222 w 122 = v 312 v 122 , and hence, each weight in the 3rd layer can be replaced by a constant or a ratio of path-values.
The weights connected with nodes O 1 j , j = 1 , 2 are rescaled, i.e., the incoming and outgoing weights of node O 1 j are multiplied and divided by the corresponding outgoing diagonal weight (i.e., w 2 jj w 3 jj ).
Afterwards, each weight in the middle layer can be replaced by a constant or a ratio of path-values, and each weight in the 1st layer can be replaced by a path-value.
Therefore, each weight in this network are replaced by a path-value or a ratio of path-values.
As for a multi-layer setting, the re-parameterization process is similar to that of a 3-layer MLP.
In particular, this process can be proceeded by rescaling weights orderly, from which is connected with the nodes in the last hidden node layer to the ﬁrst hidden node layer, and the scalar of the rescaling operation at each node, is the corresponding outgoing diagonal weight.
After such rescaling steps, the weights can be re-parameterized as follows: (1) w 1 jk → v 1 jk ; (2) w lkk → 1 ; (3) w ljk → v ljk v 1 kk , l (cid:54) = 1 , j (cid:54) =
k .
In summary, a network with weight at each edge (e.g., Step 1 in Fig. 3), can be equivalently reparameterized into another network with a path-value or a ratio of path-values at each edge (e.g.,
Step 4 in Fig. 3), by rescaling its weights layer by layer.
After the re-parameterization, the outputs of hidden nodes can be calculated by using path-values, in the same way as they are calculated by weights, and hence, Theorem 3.1 is established.
(cid:4)
Remark :
The positive constraint is not essential for proving Theorem 3.1, because when the diagonal weights are negative, the re-parameterization can still be conducted correctly by using the absolute value of the corresponding weight as the rescaling scalar.
Considering the robustness and computational efﬁciency, this constrain exists in the optimization algorithm
(Meng et al., 2019), and hence, we follow it.
Besides, this constrain will not bring much inﬂuence on the model expressiveness, because the number of the constrained weights is tiny, compared with the total weights.
Moreover, we discuss some other choices on the re-parameterization and the inﬂuence in Appendix D.
3. A NALYZING THE BN IN THE P ATH S PACE
With the re-parameterization in Section 3.1, the forward propagation can be ensured in the path space, and the outputs of hidden nodes can be calculated by using path-values.
Hence, in this subsection, we will design BN in the path space, via applying and analyzing the conventional BN to the re-parameterized network.
Firstly , we brieﬂy introduce the conventional BN.
Given a mini-batch of inputs { x i ; i = 1 , · · · , m } ,
BN can normalize each output of the hidden node as BN ( o lj ( x i ))
= γ lj · o lj ( x i ) − µ lj σ lj + β lj , where µ lj =
[Equation]
mi =1 o lj
( x i ) and σ lj = (cid:113) 1 m (cid:80) mi =1 ( o lj ( x i ) − µ lj ) 2 + (cid:15) represent the mean and standard deviation of the minibatch outputs, and γ lj , β lj are the scale and shift term 2 which are independent with w , respectively.
Then, we can denote the output after BN and ReLU as z lj ( x i ) =
g (cid:0) BN
( o lj ( x i ))
(cid:1) .
Secondly , we analyze the gradient propagation in the re-parameterized network with the the conventional BN.
Here, we use L and ∇ z l ( x i )
L = (cid:16) ∂
L ∂z l 1 ( x i ) , · · · , ∂ L ∂z ld ( x i ) (cid:17) , to denote the cross-entropy loss and the gradient vector w.r.t hidden output in layer l , respectively.
Then, we can give an estimation of the norm of ∇ z l ( x i )
L in Theorem 3.2.
Theorem 3.2 We use W (cid:48) l to denote the parameter matrix in layer l after the re-parameterization.
Suppose z lj ( x i ) <
O (1) , ∀ i, j, l , and m > 64 .
The norm of the gradient w.r.t z l ( x i ) can be upper bounded as
[Equation]
where σ l = diag ( σ l 1 , · · · , σ ld ) , γ l = diag ( γ l 1 , · · · , γ ld ) , and D l ( x i ) =
diag ( D l 1 ( x i ) , · · · , D ld ( x i )) .
Here, diag ( a 1 , · · · , a d ) represents a diagonal matrix whose diagonal elements are a 1 , · · · , a d , and D lj ( x i )
=
if z lj ( x i ) > 0 , otherwise D lj ( x i ) = 0 .
The proof of Theorem 3.2 can be found in Appendix B.
Discussion:
According to Theorem 3.2, the gradient norm exponentially depends on depth L .
If the spectral norm of matrix D s ( x i ) · γ s · ( σ s ) − 1 · W (cid:48) s roughly equals to 1 , the error signals can be effectively propagated in the backward process (i.e., not vanished or exploded).
On the one hand, if there is non-zero elements in D s ( x i ) , its spectral norm will equal to 1 .
On the other hand, in practice, γ sj is initialized to be 1 , and will keep its value around 1 during the training process, so the spectral norm of γ s will be around 1 .
Hence, the magnitudes of W (cid:48) s and σ − 1 s is crucial for the gradient norm.
For matrix W (cid:48) s , according to Theorem 3.1, the diagonal elements of W (cid:48) s
( s ≥ 2) are degenerated to be constant 1 , and these constant terms will not be trained.
Besides, the trained parameters are initialized to be much smaller than 1 (cf.
Appendix D.3), so matrix W (cid:48)
s will approach to an identity matrix.
For matrix σ − 1 s , if each σ sj is smaller than 1 , it can cause gradient exploding during the back propagation process.
We will conduct observational experiments in Section 5.2, and it shows that (cid:107) σ s (cid:107) is less than 1 and gradients becomes larger during the back propagation process.
Therefore, such discussion result motivates us to design BN in the path space by modifying the conventional BN, in order to propagate the error signals effectively.
We will next detailedly introduce the proposed BN in the path space, i.e., P -BN.
P -BN: E FFECTIVE BN IN THE P ATH S PACE
In this section, we design the BN in the path space, namely P -BN, via slightly modifying the forward process of the conventional BN.
Motivated by the calculation for the outputs of hidden node in
Theorem 3.1, as well as the discussion of Theorem 3.2, we propose to normalize the terms related to the trained coefﬁcients (i.e., Eq.
and the 2nd term in Eq. 2), and exclude the term related to the constant coefﬁcient (i.e., the 1st term in Eq. 2).
Speciﬁcally, we use z lj ( x i ) to denote the output after
P -BN and ReLU, and detailedly describe the forward process of P -BN in the following two steps.
(1) For the 1st layer, the operation of P -BN is the same as the conventional BN, i.e., z lj ( x i )
=
z lj ( x i ) .
(2) For the l-th layer ( l ≥ 2 ), as shown in Fig. 4, there are three sub-steps for P -BN as follows.
BN 𝒐𝒛
BN
Mean 𝝁 𝟐𝟐 , Variance 𝝈
BN 𝓟 -BN 𝒛𝒐 𝟐𝟐,/
Mean 𝝁 𝟐𝟐,/ ,
Variance 𝝈 𝟐𝟐,/ 𝒗𝒗𝒗𝒗
First , calculate the partial weighted summation.
Here, the input related to the diagonal constant element in the parameter matrix is excluded, i.e.,
[Equation]
Second , normalize the partial weighted summation above, i.e.,
[Equation]
Third , add the excluded term z l − 1 j ( x i ) into the normalized partial weighted summation in Eq.
4, i.e.,
[Equation]
Consequently, P -BN can be easily combined with the optimization algorithm in the path space for ReLU networks, and we will show it in Algorithm 1 in
Appendix E.
Then, we will show that the gradients can be effectively propagated by using P -BN in Theorem 4.1.
Theorem 4.1 Suppose z lj ( x i ) <
O (1) , ∀ i, j, l , and m > 64 , the norm of gradient w.r.t z l ( x i ) can be upper bounded as
[Equation]
where ˆ W (cid:48) s = W (cid:48)
s − I , σ l,/ = diag ( σ l,/ 1 , · · · , σ l,/d ) , γ l,/ = diag ( γ l,/ 1 , · · · , γ l,/d ) and D l ( x i ) =
diag ( D l 1 ( x i ) , · · · , D l d ( x i ))
where D l j ( x i )
=
if z lj ( x i ) > 0 , otherwise D l j ( x i )
= 0 .
The proof of Theorem 4.1 can be found in Appendix B.
Then, we give Corollary 4.2, via summarizing from Theorem 3.2 and Theorem 4.1. Corollary 4.2 Using the notations in Theorem 3.2 and Theorem 4.1 and initializing γ l and γ l,/ to be identity matrix, we have the following upper bound for gradient norm at initialization point [Equation] [Table]
[Figure] [Figure] [Figure] Suppose λ = (cid:107) ˆ W (cid:48) s (cid:107) .
A sufﬁcient condition for that (cid:107)∇ z l ( x i ) L(cid:107) and (cid:107)∇ z l ( x i ) L(cid:107) can be upper bounded by constant (i
[Figure]
e., will not diverge as L → ∞ ) is σ sj ∈ [ L (1+ λ ) L +1 , L (1+ λ )
L − 1 ] and σ s,/j ∈ [ λL, ∞ ) ∀ j , respectively. Discussion: We can conclude from Corollary 4.2 that the range for σ s,/j is much larger than that for σ sj . Furthermore, if λ ≤ 1 L , we have [ L (1+ λ ) L +1 , L (1+ λ ) L − 1 ] ⊂ [ λL, ∞ ) (i.e., the range for σ sj is fully contained in the range σ s,/j ), which indicates the backward stability of P -BN. We will observe the variance in each layer in Section 5.2, and the results demonstrate that the variance becomes larger with P -BN.
Thus, to some extent, the gradient propagated more stable for NN with P -BN.
E XPERIMENTS
In this section, we conduct experiments by ﬁrst comparing the performance of P -BN with the baselines on various datasets with some network structures, and then showing some experimental observations to support theoretical analyses.
5. P ERFORMANCE E XPERIMENTS
In this subsection, we evaluate the performance of P-BN on training deep neural networks by conducting experiments on three datasets, CIFAR-10, CIFAR-100, and ImageNet.
We will next introduce the network structures and compared methods ﬁrst.
5.1. N ETWORK S TRUCTURES AND C OMPARED M ETHODS
In this subsection, we describe the network structures and compared methods.
First , as for the network structures, we apply P -BN to train ResNet and PlainNet (He et al., 2016).
Second , we show the setting details of four compared methods as follows
(1) SGD+BN:
We use SGD to train the network with the conventional BN.
(2) G -SGD+BN:
We use G -SGD to train the network with the conventional BN.
(3) G -SGD+BN(wnorm): We use a method which was intuitively proposed to handle the conventional BN without any theoretical analysis (Meng et al., 2019).
Here, the gradient of the path-value,
(4) G -SGD+ P -BN:
Our proposed P -BN targets on BN in the path space, and the network with P -BN should be optimized by the algorithms in the path space.
Thus, we use G -SGD to train the network with P -BN.
5.1. CIFAR
[Table] We conduct experiments on CIFAR-10 dataset and CIFAR-100 dataset (Krizhevsky et al., 2009).
Here, we train ResNet of 18, 34, and 50 layers, and PlainNet of 18 and 34 layers, respectively.
As for the hyper-parameters of methods mentioned in their corresponding papers, i.e., SGD+BN (He et al., 2016) and G -SGD+BN(wnorm)
(Meng et al., 2019), we use the same settings as their original ones.
Besides, we tune the hyper-parameters for other methods, i.e., G -SGD+BN, and the proposed G -SGD+ P -BN.
More details about the experimental setting can refer to Appendix F.1.
In Table 1, we show the performance results on the test error rate, and in Fig.
8, we plot the training curve and test accuracy of 50-layer ResNet.
Such results demonstrate that: (1) G -SGD+ P -BN outperforms others on all tested datasets and network structures, which shows the superiority of P -BN; (2) Combining the path space and the conventional BN directly hurts performance, which empirically motivates us to propose P -BN; (3) G -SGD+ P -BN outperforms G -SGD+BN(wnorm), which shows the beneﬁt and signiﬁcance of our proposed formal analyses.
5.1. I MAGE N ET
We conduct experiments on ImageNet dataset (Russakovsky et al., 2015).
Here, we train ResNet of 50, 101 and 152 layers, to demonstrate that the proposed P -BN can help to train very deep networks on a large dataset, and we compare the performance results between SGD+BN and G -SGD+ P -BN.
The method for tuning the hyper-parameters is similar to that in Section 5.1.2, and more details can refer to Appendix F.1.
In Table 2, we list top-1 and top-5 test error rates, and in Fig.
5, plot the training curve and test accuracy during training ResNet of 50, 101, and 152 layers.
These results well demonstrate the superiority of G -SGD+ P -BN, and show that for the large dataset and very deep networks, P -BN can still outperform the conventional BN.
5. O BSERVATIONS ON S TANDARD D EVIATIONS AND G RADIENTS
After analyzing the advantage of P -BN over the conventional BN theoretically in Section 3.2 and 4, we now provide some experimental observations to support our analyses.
Here, we will show the magnitude of standard deviations in BN, and the gradients of ReLU networks with BN and P -BN.
In this experiments, we use a stacked CNN of 50 layers with 128 channels in each layer, and use
CIFAR-10 as training data, with the batch size of 128.
Then, we add the conventional BN or P -BN at the end of each hidden layer in the stacked CNN, which is denoted as BN and P -BN , respectively.
For the model with the conventional BN, it is randomly initialized.
For the model with P -BN, the diagonal elements in parameter matrices (expect for the 1st layer) are initialized to 1 (cf.
Section 3.1), and other elements are initialized with small random values (cf.
Appendix D.3).
Because the theoretical analyses are not related to the training process, we will just observe some quantities at initialization as follows.
First , we show some observations on standard deviation σ lj in Theorem 3.2.
Here, we log σ lj for each hidden node during the forward process of the 1st mini-batch, and then calculate the average standard deviation among the nodes in each layer.
In Fig.
6, we plot the curves regarding standard deviations, and we can observe that: (1) For the conventional BN, the standard deviations are smaller than 1 , which matches the claim in Theorem 3.2, and is one of the reasons of the gradient exploding.
(2) For P -BN, the standard deviation grows when the layer index becomes larger, and most of them are larger than 1 .
Thus, in Theorem 4.1, (cid:107) σ − 1 s · W (cid:48) s (cid:107) are smaller than (cid:107) I (cid:107) , and (cid:107)∇ z l L(cid:107) can be upper bounded, which is one of the reasons for that the gradient exploding can be weaken by P -BN.
Second , we show some observations on gradients, to investigate that whether the gradient exploding can be weaken by P -BN in practice.
Here, we log the L2-norm of gradients of loss w.r.t.
the outputs in each layer, and plot the log of the gradient norm in Fig.
We can observe that, the gradients indeed explode in the ReLU network with both the conventional BN and P -BN, and the exploding speed is much slower with P -BN, which demonstrates that P -BN can help to weaken the gradient exploding.
C ONCLUSION
In this paper, we conduct a formal study on the design of BN in the path space.
First, considering that BN is adopted during the forward process, we ensure the forward propagation in the path space, via proposing a re-parameterization for weights in ReLU networks.
Here, the re-parameterization can equivalently replace each weight in the original network with a new value calculated by one or several path-values, and hence, the outputs of hidden nodes can be calculated by using path-values.
Then, we design BN in the path space, by applying and analyzing the conventional BN to the reparameterized network.
Here, we analyze the gradient propagation after BN is applied, and then notice that the gradient will explode in such networks.
Thus, we propose a novel BN in the path space, i.e., P -BN, which excludes the term whose coefﬁcient is ﬁxed by re-parameterization from normalization.
Finally, we conduct experiments to verify the proposed method, and results on both
CIFAR and ImageNet dataset, demonstrate that P -BN can enhance the performance signiﬁcantly.
A N OTATIONS
Notation
Object
[Table]
Weight
[Table]
edge which connects
[Table]
k -th
[Table]
l
[Table] [Table]
layer
[Table]
j
[Table]
th node
[Table]
l -th layer
[Table]
only contains
[Table]
elements
[Table]
matrices
[Table]
i.e.
[Table]
Ll
[Table]
diagonal
[Table]
i
[Table]
e.
[Table]
v
[Table]
l
[Table] [Table]
[Table] [Table]
B S UPPLEMENTARY P ROOFS
Theorem B.
We use W (cid:48) l to denote the parameter matrix in layer l after the re-parameterization.
Suppose z lj ( x i ) <
O (1) , ∀ i, j, l , and the batch size m > 64 .
The norm of gradient w.r.t z l ( x i ) can be upper bounded as
[Equation]
where σ l = diag ( σ l 1 , · · · , σ ld ) , γ l = diag ( γ l 1 , · · · , γ ld ) and D l ( x i ) =
diag ( D l 1 ( x i ) , · · · , D ld ( x i )) .
Here, diag ( a 1 , · · · , a d ) represents a diagonal matrix whose diagonal elements are a 1 , · · · , a d , and D lj ( x i )
=
if z lj ( x i ) > 0 , otherwise D lj ( x i ) = 0 .
Proof: We concat the gradient vector of different instances in a minibatch into one ( d × m ) - dimensional vector as
[Equation]
and hence, according to chain rule, we have
[Equation]
Here, on the one hand, as for matrix ∂z s ∂o s , it donates ∂z s
[Equation]
 .
So we have
[Equation]
Obviously, most of the elements in the matrix equals to zero.
Under the assumption that z sj ( x i ) <
O (1) for all i, j, l and m > 64 , the above matrix approaches to the diagonal matrix D s · γ s · ( σ s ) − 1 .
On the other hand, as for matrix ∂o s ∂z s − 1 , it donates
[Equation]
Thus, using the fact that (cid:107) AB (cid:107) ≤ (cid:107) A (cid:107)(cid:107) B (cid:107) , we can get the result in the theorem.
[Equation]
Theorem B.2 Suppose z lj ( x i ) <
O (1) , ∀ i, j, l , and m > 64 .
The norm of gradient w.r.t z l ( x i ) can be upper bounded as
[Equation]
L(cid:107) ≤ O (cid:32) L (cid:89) s
=
l + (cid:107) D s ( x i ) · ( I + γ s,/ · ( σ s,/ ) − 1 · ˆ W (cid:48)
s ) (cid:107) (cid:33) , where ˆ W (cid:48) s = W (cid:48)
s − I , σ l,/ = diag ( σ l,/ 1 , · · · , σ l,/d ) , γ l,/ = diag ( γ l,/ 1 , · · · , γ l,/d ) and D l ( x i ) =
diag ( D l 1 ( x i ) , · · · , D l d ( x i ))
where D l j ( x i )
=
if z lj ( x i ) > 0 , otherwise D l j ( x i )
= 0 .
Proof: According to the chain rule, we have
[Equation]
(cid:33)
[Equation]
13) where ∂z s ∂o s,/ denotes
[Equation]
s
[Equation]
x
[Equation]
∂z sd [Equation] x
[Equation]
∂o s
[Equation] [Equation]
· · · ∂z
[Equation]
x
[Equation]
∂o
[Equation]
[Equation]


[Equation]
So we have
[Equation]
i
[Equation]
Obviously, most of the elements in the matrix equals to zero.
Under the assumption that z lj ( x i ) is bounded for all i, j, l , the above matrix tend to be the diagonal matrix D l · γ l · ( σ l ) −
if m approaches ∞ .
On the one hand, as for matrix ∂o s,/ ∂z s − 1 , we have
[Equation]
On the other hand, as for matrix ∂o s,/ ∂z s
[Equation]
1 , we have
[Equation]
Thus, using the fact that (cid:107) AB (cid:107) ≤ (cid:107) A (cid:107)(cid:107) B (cid:107) , we can get the result in the theorem.
(cid:4)
C D ETAILS ON O PTIMIZATION A LGORITHMS IN THE P ATH S PACE
We provide the detailed update rules on G -SGD (Meng et al., 2019) in this section.
Here, we use w j and w i to denote the diagonal weight and non-diagonal weight at layer 1 , respectively, and use w j (cid:48) and w i (cid:48) to denote the diagonal weight and non-diagonal weight at layer l , l ≥ 2 , respectively.
The gradient of w w.r.t loss l , can be represented by δ w .
Then we have
[Equation]
where event w j → w i (cid:48) , means w j and w i (cid:48) are contained in one of the paths, containing one nondiagonal weights at most.
D D ISCUSSION ABOUT R E - PARAMETERIZATION D. U NEQUAL
[Figure]
IDTH
[Figure]
In this section, we ﬁrst show the method for ﬁnding weights used for rescaling in Theorem 3.1, which is denoted as “red weights” here, for the network with unequal width of the input and output layers.
As shown in Fig.
7, for weight matrix { w l ; l = 2 , · · · , L − 1 } , the diagonal elements are selected as red weights.
For weight matrix w 1 and w L with size ( D, h 1 ) and ( h L − 1 , K ) , where
D and K are number of input and output nodes, respectively, and h i is the number of nodes in the i-th hidden layer, then, elements { w 1 ( i mod D, i ); i = 1 , · · · , h 1 } and { w L ( i, i mod K ); i = 1 , · · · , h L − 1 } are selected as red weights, respectively.
Besides, all methods can be easily applied to CNN.
As we all know, a ﬁlter in CNN transforms a input feature map to the output feature map.
Thus, each feature map can be regarded as a hidden node in MLP, and then the ﬁlter connects two feature maps in CNN corresponds to the weight connects two hidden nodes in MLP.
Besides, the method to identify red weights is similar with that for MLP unless the diagonal elements of the 4-triple of the weight matrices are selected.
Suppose the 4-triple of weight matrices is [64 , 64 , 3 , 3] where the ﬁrst two elements denote the number of input channels and output channels, the last two elements denote the size of ﬁlter.
The element [0 , 0 , 1 , 1] , [1 , 1 , 1 , 1] , [2 , 2 , 1 , 1] , · · · , [63 , 63 , 1 , 1] are selected to be red weights.
In this way, the re-parameterization method in Theorem 3.1 is easy to extend to CNN by rescaling the weights using the selected red weights.
D. O THER C HOICES ON R E - PARAMETERIZATION
In this section, we discuss other choices on re-parameterization and the inﬂuence.
The reparameterization method using path-values showed in the proof of Theorem 3.1 is not unique.
For example, we can obtain another kind of re-parameterization, via multiplying and dividing the incoming and outgoing weights of O 11 by v 111 in step 4 of Figure.3.
This will not inﬂuence analyses much in this paper for the following two reasons.
First, whatever re-parameterization method used, each reparameterized network can serve as a sufﬁcient representation for ReLU networks in the path space, because the outputs of the network will keep unchanged for any input after re-parameterization.
Second, our studies are based on Theorem 3.1, which can be generalized to other re-parameterization methods.
D. R ELATIONSHIP BETWEEN R E - PARAMETERIZATION AND O PTIMIZATION P ROCESS
In this section, we show how the network should be optimized after it has been re-parameterized.
Speciﬁcally, after re-parameterization, we can optimize the loss function according to gradients with respect to the path-values using the optimization algorithm such as SGD as follows.
First, we denote the parameter matrix of the re-parameterized network at the l-th layer as W (cid:48) l , whose elements are constant, path-value, and ration of path-values.
Next, we can use back propagation to obtain the gradient of W (cid:48) l .
According to the chain rule, for the path-values that will not appear in the denominator, its gradient can be calculated as:
[Equation]
and for the path-values that will appear in the denominator, its gradient can be calculated as:
[Equation]
Then, we can derive the update rule of SGD in path space mentioned in Section C. Here, please note that the update rules derived from re-parameterization is consistent with that provided in (Meng et al., 2019), and re-parameterization provides a much simpler and straightforward way to explain the complex update rules.
Based on the process above, we now provide some analyses on the magnitude of parameter matrices.
Speciﬁcally, for the 2nd term in Eq. 23, the numerator contains many terms, which tends to be larger than the gradient of other paths, and thus, the denominator, i.e., v 1 kk , ( k = 1 , · · · , d ) is initialized to be large in practice to match the magnitude of the numerator.
Therefore, for the parameter matrix W (cid:48) l , v 1 kk , ( k = 1 , · · · , d ) appears in the denominator for most of its elements, and hence, these elements are initialized to be much smaller than 1 , which supports analyses in Theorem 3.2.
E F ULL A LGORITHM FOR O PTIMIZING R E LU N ETWORKS WITH P -BN
In Algorithm 1, we show the full algorithm for optimizing ReLU networks with P -BN in the path space.
Besides, please check more details of PathOptimizer in Section 2.2 and Appendix C.
F D ETAILS ON E XPERIMENTS F. E XPERIMENTAL S ETTING D ETAILS
In particular, following the settings in Meng et al.
(2019), we apply G -SGD or P -BN with residual blocks, because there is no positive scaling invariance across residual blocks.
In addition, we introduce a coefﬁcient λ ( λ < 1) for z l − 1 j ( x i ) in Eq. 5 for experiments of ResNet, to prevent the output exploding during the forward process, and λ for all layers in ResNet is set to be 0.1. Update Rule of Path Optimizer Ensure: w ( T ) .
Here we describe pre-processing steps, which is same as He et al.
For CIFAR-10 and CIFAR100, we randomly crop the input image by size of 32 with padding size of 4, and normalize every pixel value to [0 , 1] .
Then the random horizontal ﬂipping to the image is applied.
For ImageNet, we randomly crop the input image by size of 224, and normalize every pixel value to [0 , 1] .
Then the random horizontal ﬂipping to the image is also applied.
We use 1 NVIDIA Tesla P100 or P40 GPU to run the experiments on CIFAR, and use 4 NVIDIA
Tesla P100 or P40 GPUs in one machine to run the experiments on ImageNet.
All experiments are averaged over 5 runs with different random seeds, and PyTorch (Paszke et al.) is used for implementation.
Here we list hyper-parameters used for CIFAR-10 and CIFAR-100.
As for the hyper-parameters of methods mentioned in their corresponding papers, i.e., SGD+BN (He et al., 2016) and ( GSGD)+BN(wnorm) (Meng et al., 2019), we use the same settings as their original ones.
Speciﬁcally, for SGD+BN, the initial learning rate is set to 0.1, and for G -SGD+BN (wnorm), the initial learning rate is set to 1.0.
Besides, we tune hyper-parameters for ( G -SGD)+BN and the proposed ( G -SGD)+( P -BN).
Speciﬁcally, the initial learning rate is searched from { 0 . 1 , 0 . 2 , 0 . 5 , 1 . 0 } .
Then, we use the method proposed in (Zheng et al., 2018) as the weight decay in the path space, and set the coefﬁcient of the weight decay to 1 × 10 − 4 , for these two methods.
Besides, we use the SGD without momentum in our experiments, because the way to utilize momentum in the path space remains unclear now.
Moreover, for all models and algorithms, the mini-batch size is set to be 128 and the training process is conducted for 64k iterations.
The learning rates are multiplied by 0.1 after 32k and 48k iterations in all experiments, and the coefﬁcient of weight decay for methods is set to be 0.0001.
Here we list hyper-parameters used for ImageNet.
For all experiments here, the mini-batch size is set to be 256 and the hyper-parameter for weight decay is set to be 0.0001 He et al.
For
SGD+BN, the initial learning rate is set to 0.1 He et al.
For G -SGD+ P -BN, we only tune the initial learning rate from { 0 . 1 , 0 . 2 , 0 . 5 } .
The training process is conducted for 90 epochs, and the learning rate is multiplied by 0.1 after 30 and 60 epochs.
As for the stacked CNN used for observations in Section 5.2, we use the following settings: kernel size 3, stride 1, padding 1, and no bias.
For CNN, the network width is the the number of channels, which is set to 128 for all hidden layers.
F. S UPPLEMENTARY R ESULTS
In Fig. 8, plot the training curve and test accuracy of 50-layer ResNet.
Here, we add other baseline, i.e., SGD+ P -BN, in which we use SGD to train the network with P -BN, and in Table 4, we show [Figure] [Figure] the test error for SGD+ P -BN.
We can observe that G -SGD+ P -BN outperforms SGD+ P -BN, which demonstrate the beneﬁts of optimizing in the path space.
F. E XPERIMENTS FOR P ATH -SGD
As mentioned in Section 2.
[Figure]
Path-SGD (Neyshabur et al., 2015a) is another PSI-related optimization algorithm, but not the optimizer in the path space.
[Table]
First , we introduce the network structure and some basic settings.
Here, because Path-SGD is computationally cost for CNN, we follow the experimental settings in (Neyshabur et al., 2015a), and use an
MLP of 5 layers, whose number hidden nodes in each layer is 4000, and the conventional BN or PBN is added at the end of every hidden layer.
Besides, we use CIFAR-10 as the training data, and the batch size is set to 64.
Moreover, all experiments are run for 150 epochs, and results are obtained by averaging over 5 runs.
The only hyper-parameter needs to be tuned is learning rate, which is searched from { 1 , 5 } −{ 1 , 2 , 3 } .
Then , we show the setting details of 4 compared methods as follow:
• SGD+BN:
We use SGD to train the network with the conventional BN.
• Path-SGD: We use Path-SGD to train the network without BN.• Path-SGD+BN:
We use Path-SGD to train the network with the conventional BN.
• Path-SGD+ P -BN: We use Path-SGD to train the network with P -BN.
In Table 5, we list the validation error of these 4 methods.
Results show that: (1) Path-SGD+ P -BN outperforms other baselines, which well demonstrate the superiority of P -BN.
(2) Path-SGD+ P -BN and Path-SGD+BN perform better than SGD+BN, which show the beneﬁts of optimizing in the path space.
F. A NALYSIS OF C OMPUTATIONAL C OSTS
Different from the conventional BN, P -BN is to identify the diagonal elements of weight matrices.
For convolutional networks, the outputs of hidden nodes in MLP always correspond to the feature maps.
Hence, on the one hand, if the sizes of feature maps in different layers are the same, and the stride length is 1 , we can then set the diagonal elements in weight matrices to 0 and add skip connections for all layers (except the 1st layer), which will not bring extra computational cost.
On the other hand, if the stride length is larger than 1 , we can also set the diagonal elements in weight matrices to 0 , and additionally, use a ﬁxed-weight convolutional layer, with only diagonal elements ﬁxed to 1 and others ﬁxed to 0 .
Thus, with the skip connection or the ﬁxed-weight convolutional layer, the back propagation process can be easily implemented.
Therefore, according to the above implementation, P -BN will not bring much extra computational cost compared with the conventional BN.
Statistically, in our experiments, the ratio of run time between G -SGD+Path-BN and SGD+BN, is less than 4 : 3 , and the extra computational cost is mainly brought by the update of G -SGD instead of P -BN.