Original: L INEARLY C ONSTRAINED W EIGHTS : R ESOLVING THE V ANISHING G RADIENT P ROBLEM BY R EDUCING A NGLE B IAS
Revised: L INEARLY C ONSTRAINED W EIGHTS : R ESOLVING THE V ANISHING G RADIENT P ROBLEM BY R EDUCING A NGLE B IAS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In this paper, we ﬁrst identify angle bias , a simple but remarkable phenomenon that causes the vanishing gradient problem in a multilayer perceptron (MLP) with sigmoid activation functions.
Revised: In this paper, we ﬁrst identify angle bias , a simple but remarkable phenomenon that causes the vanishing gradient problem in a multilayer perceptron (MLP) with sigmoid activation functions.

Original: We then propose linearly constrained weights (LCW) to reduce the angle bias in a neural network, so as to train the network under the constraints that the sum of the elements of each weight vector is zero.
Revised: We then propose linearly constrained weights (LCW) to reduce the angle bias in a neural network, so as to train the network under the constraints that the sum of the elements of each weight vector is zero.

Original: A reparameterization technique is presented to efﬁciently train a model with LCW by embedding the constraints on weight vectors into the structure of the network.
Revised: A reparameterization technique is presented to efﬁciently train a model with LCW by embedding the constraints on weight vectors into the structure of the network.

Original: Interestingly, batch normalization (Ioffe & Szegedy, 2015) can be viewed as a mechanism to correct angle bias.
Revised: Interestingly, batch normalization (Ioffe & Szegedy, 2015) can be viewed as a mechanism to correct angle bias.

Original: Preliminary experiments show that LCW helps train a 100-layered MLP more efﬁciently than does batch normalization.
Revised: Preliminary experiments show that LCW helps train a 100-layered MLP more efﬁciently than does batch normalization.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural networks with a single hidden layer have been shown to be universal approximators (Hornik et al., 1989; Irie & Miyake, 1988).
Revised: Neural networks with a single hidden layer have been shown to be universal approximators ( Hornik et al. , 1989; Irie & Miyake, 1988).

Original: However, an exponential number of neurons may be necessary to approximate complex functions.
Revised: However, an exponential number of neurons may be necessary to approximate complex functions.

Original: A solution to this problem is to use more hidden layers.
Revised: A solution to this problem is to use more hidden layers.

Original: The representation power of a network increases exponentially with the addition of layers ( Telgarsky, 2016; Eldan & Shamir, 2016).
Revised: The representation power of a network increases exponentially with the addition of layers (Telgarsky, 2016; Eldan & Shamir, 2016).

Original: A major obstacle in training deep nets, that is, neural networks with many hidden layers, is the vanishing gradient problem.
Revised: A major obstacle in training deep nets, that is, neural networks with many hidden layers, is the vanishing gradient problem.

Original: Various techniques have been proposed for training deep nets, such as layer-wise pretraining ( Hinton & Salakhutdinov, 2006), rectiﬁed linear units (Nair & Hinton, 2010; Jarrett et al., 2009), variance-preserving initialization (Glorot & Bengio, 2010), and normalization layers (Ioffe & Szegedy, 2015; G ¨ ulc¸ehre & Bengio, 2016).
Revised: Various techniques have been proposed for training deep nets, such as layer-wise pretraining (Hinton & Salakhutdinov, 2006), rectiﬁed linear units ( Nair & Hinton, 2010; Jarrett et al., 2009), variance-preserving initialization (Glorot & Bengio, 2010), and normalization layers (Ioffe & Szegedy, 2015; G ¨ ulc¸ehre & Bengio, 2016).

Original: In this paper, we ﬁrst identify the angle bias that arises in the dot product of a nonzero vector and a random vector.
Revised: In this paper, we ﬁrst identify the angle bias that arises in the dot product of a nonzero vector and a random vector.

Original: The mean of the dot product depends on the angle between the nonzero vector and the mean vector of the random vector.
Revised: The mean of the dot product depends on the angle between the nonzero vector and the mean vector of the random vector.

Original: We show that this simple phenomenon is a key cause of the vanishing gradient in a multilayer perceptron (MLP) with sigmoid activation functions.
Revised: We show that this simple phenomenon is a key cause of the vanishing gradient in a multilayer perceptron (MLP) with sigmoid activation functions.

Original: We then propose the use of so-called linearly constrained weights (LCW) to reduce the angle bias in a neural network.
Revised: We then propose the use of so-called linearly constrained weights (LCW) to reduce the angle bias in a neural network.

Original: LCW is a weight vector subject to the constraint that the sum of its elements is zero.
Revised: LCW is a weight vector subject to the constraint that the sum of its elements is zero.

Original: A reparameterization technique is presented to embed the constraints on weight vectors into the structure of a neural network.
Revised: A reparameterization technique is presented to embed the constraints on weight vectors into the structure of a neural network.

Original: This enables us to train a neural network with LCW by using optimization solvers for unconstrained problems, such as stochastic gradient descent.
Revised: This enables us to train a neural network with LCW by using optimization solvers for unconstrained problems, such as stochastic gradient descent.

Original: Preliminary experiments show that we can train a 100-layered MLP with sigmoid activation functions by reducing the angle bias in the network.
Revised: Preliminary experiments show that we can train a 100-layered MLP with sigmoid activation functions by reducing the angle bias in the network.

Original: Interestingly, batch normalization (Ioffe & Szegedy, 2015) can be viewed as a mechanism to correct angle bias in a neural network, although it was originally developed to overcome another problem, that is, the internal covariate shift problem.
Revised: Interestingly, batch normalization (Ioffe & Szegedy, 2015) can be viewed as a mechanism to correct angle bias in a neural network, although it was originally developed to overcome another problem, that is, the internal covariate shift problem.

Original: Preliminary experiments suggest that LCW helps train deep MLPs more efﬁciently than does batch normalization.
Revised: Preliminary experiments suggest that LCW helps train deep MLPs more efﬁciently than does batch normalization.

Original: In Section 2, we deﬁne angle bias and discuss its relation to the vanishing gradient problem.
Revised: In Section 2, we deﬁne angle bias and discuss its relation to the vanishing gradient problem.

Original: In
Revised: In

Original: Section 3, we propose LCW as an approach to reduce angle bias in a neural network.
Revised: Section 3, we propose LCW as an approach to reduce angle bias in a neural network.

Original: We also present a reparameterization technique to efﬁciently train a model with LCW and an initialization method for LCW.
Revised: We also present a reparameterization technique to efﬁciently train a model with LCW and an initialization method for LCW.

Original: In Section 4, we review related work; mainly, we examine existing normalization techniques from the viewpoint of reducing the angle bias.
Revised: In Section 4, we review related work; mainly, we examine existing normalization techniques from the viewpoint of reducing the angle bias.

Original: In Section 5, we present empirical resultsthat show that it is possible to efﬁciently train a 100-layered MLP by reducing the angle bias using
Revised: In Section 5, we present empirical results

Original: LCW.
Revised: [Figure]

Original: Finally, we conclude with a discussion of future works.
Revised: i n d

Original: [Figure]
Revised: [Figure]

Revised: (c) Multiplication WA .

Original:  A NGLE BIAS
Revised:  A NGLE BIAS

Original: We introduce angle bias by using the simple example shown in Figure 1.
Revised: We introduce angle bias by using the simple example shown in Figure 1.

Original: Figure 1(a) is a heat map representation of matrix W 2 R 100 (cid:2) 100 , each of whose elements is independently drawn from a uniform random distribution in the range ( (cid:0) 1 ; 1) .
Revised: Figure 1(a) is a heat map representation of matrix W 2 R 100 (cid:2) 100 , each of whose elements is independently drawn from a uniform random distribution in the range ( (cid:0) 1 ; 1) .

Original: Matrix A 2 R 100 (cid:2) 100 is also generated randomly, and its elements range from 0 to 1, as shown in Figure 1(b).
Revised: Matrix A 2 R 100 (cid:2) 100 is also generated randomly, and its elements range from 0 to 1, as shown in Figure 1(b).

Original: We multiply W and A to obtain the matrix shown in Figure 1(c).
Revised: We multiply W and A to obtain the matrix shown in Figure 1(c).

Original: Unexpectedly, a horizontal stripe pattern appears in the heat map of W A although both W and A are random matrices.
Revised: Unexpectedly, a horizontal stripe pattern appears in the heat map of W A although both W and A are random matrices.

Original: This pattern is attributed to the angle bias that is deﬁned as follows:
Revised: This pattern is attributed to the angle bias that is deﬁned as follows:

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: P (cid:13) is an m dimensional probability distribution whose expected value is (cid:13) 1 m , where (cid:13) 2 R and 1 m is an m dimensional vector whose elements are all one.
Revised: P (cid:13) is an m dimensional probability distribution whose expected value is (cid:13) 1 m , where (cid:13) 2 R and 1 m is an m dimensional vector whose elements are all one.

Original: Proposition 1.
Revised: Proposition 1.

Original: Let a be a random vector in R m that follows P (cid:13) .
Revised: Let a be a random vector in R m that follows P (cid:13) .

Original: Given w 2 R m such that k w k > 0 , the expected value of w (cid:1) a is j (cid:13) jp m k w k cos (cid:18) w , where (cid:18) w is the angle between w and 1 m .
Revised: Given w 2 R m such that k w k > 0 , the expected value of w (cid:1) a is j (cid:13) jp m k w k cos (cid:18) w , where (cid:18) w is the angle between w and 1 m .

Original: Proof.
Revised: Proof.

Original: It follows from E ( w (cid:1) a ) =
Revised: It follows from E ( w (cid:1) a ) =

Original: w (cid:1) E ( a ) =
Revised: w (cid:1) [Equation]

Original: w (cid:1) ( (cid:13) 1 m ) =
Revised: jp

Original: k w k k (cid:13) 1 m k cos (cid:18) w = j (cid:13) jp m k w k cos (cid:18) w , where E ( x ) denotes the expected value of random variable x .
Revised: [Equation] w , where E ( x ) denotes the expected value of random variable x .

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: From Proposition 1, the expected value of w (cid:1) a depends on (cid:18) w as long as (cid:13) 6 = 0 .
Revised: From Proposition 1, the expected value of w (cid:1) a depends on (cid:18) w as long as (cid:13) 6 = 0 .

Original: The distribution of w (cid:1) a is then biased depending on (cid:18) w ; this is called angle bias .
Revised: The distribution of w (cid:1) a is then biased depending on (cid:18) w ; this is called angle bias .

Original: In Figure 1, if we denote the i -th row vector of W and the j -th column vector of A by w i and a j , respectively, a j follows P (cid:13) with (cid:13) = 0 : 5 .
Revised: In Figure 1, if we denote the i -th row vector of W and the j -th column vector of A by w i and a j , respectively, a j follows P (cid:13) with (cid:13) = 0 : 5 .

Original: The i -th row of W A is biased according to the angle between w i and 1 m , because the ( i; j ) -th element of W A is the dot product of w i and a j .
Revised: The i -th row of W A is biased according to the angle between w i and 1 m , because the ( i; j ) -th element of W A is the dot product of w i and a j .

Original: Note that if the random matrix A has both positive and negative elements, W A also shows a stripe pattern as long as each column vector of A follows P (cid:13) with (cid:13) 6 = 0 .
Revised: Note that if the random matrix A has both positive and negative elements, W A also shows a stripe pattern as long as each column vector of A follows P (cid:13) with (cid:13) 6 = 0 .

Original: We can generalize Proposition 1 for any m dimensional distribution ^ P , instead of P (cid:13) , as follows:
Revised: We can generalize Proposition 1 for any m dimensional distribution ^ P , instead of P (cid:13) , as follows:

Original: Proposition 2.
Revised: Proposition 2.

Original: Let ^ a be a random vector that follows an m dimensional probability distribution ^ P whose expected value is ^ (cid:22) 2 R m .
Revised: Let ^ a be a random vector that follows an m dimensional probability distribution ^ P whose expected value is ^ (cid:22) 2 R m .

Original: Given w 2 R m such that k w k > 0 , it follows that
Revised: Given w 2 R m such that k w k > 0 , it follows that

Original: [Equation]
Revised: [Equation]

Original: if k ^ (cid:22) k > 0 , otherwise ; where ^ (cid:18) w is the angle between w and ^ (cid:22) .
Revised: where ^ (cid:18) w is the angle between w and ^ (cid:22) .

Original: Proof.
Revised: Proof.

Original: The proof is the same as that of Proposition 1.
Revised: The proof is the same as that of Proposition 1.

Original: Proposition 2 states that the distribution of w (cid:1) ^ a is biased according to ^ (cid:18) w unless k ^ (cid:22) k
Revised: Proposition 2 states that the distribution of w (cid:1) ^ a is biased according to ^ (cid:18) w unless k ^ (cid:22) k

Original: = 0 .
Revised: = 0 .

Original: 2. A NGLE BIAS IN A MULTILAYER PERCEPTRON
Revised: 2. A NGLE BIAS IN A MULTILAYER PERCEPTRON

Original: We consider a standard MLP.
Revised: We consider a standard MLP.

Original: For simplicity, the number of neurons m  vector of the i -th neuron in layer l is denoted by w li 2 R m .
Revised: For simplicity, the number of neurons m is assumed to be the same in all layers. The activation vector in layer l is denoted by a l = ( a l 1 ; : : : ; a lm ) > 2 R m . The weight vector of the i -th neuron in layer l is denoted by w l i 2 R m .

Original: It is generally assumed that k w li k > 0 .
Revised: It is generally assumed that k w l i k > 0 .

Original: The activation of the i -th neuron in layer l is given by
Revised: The activation of the i -th neuron in layer l is given by

Original: [Equation]
Revised: [Equation]

Original: where f is a nonlinear activation function, b li 2 R is the bias term, z li 2 R denotes the preactivation value, and (cid:18) li 2 R means the angle between w li and a l (cid:0) 1 .
Revised: where f is a nonlinear activation function, b li 2 R is the bias term, z li 2 R denotes the preactivation value, and (cid:18) li 2 R means the angle between w li and a l (cid:0) 1 .

Original: We assume that a 0 corresponds to the input vector to the MLP.
Revised: We assume that a 0 corresponds to the input vector to the MLP.

Original: In Equations (1), (2), and (3), variables z li , a li , and (cid:18) li are regarded as random variables whose distributions are determined by the distribution of the input vector a 0 , given the weight vectors and the bias terms.
Revised: In Equations (1), (2), and (3), variables z li , a li , and (cid:18) li are regarded as random variables whose distributions are determined by the distribution of the input vector a 0 , given the weight vectors and the bias terms.

Original: From Proposition 2 and Equation (2), if a l (cid:0) 1 follows ^
Revised: From Proposition 2 and Equation (2), if a l (cid:0) 1 follows ^

Original: P with the mean vector ^ (cid:22) l (cid:0) 1 such that k ^ (cid:22) l (cid:0) 1 k > 0 , the preactivation z li is biased according to the angle between w li and ^ (cid:22) l (cid:0) 1 .
Revised: P with the mean vector ^ (cid:22) l (cid:0) 1 such that k ^ (cid:22) l (cid:0) 1 k > 0 , the preactivation z li is biased according to the angle between w li and ^ (cid:22) l (cid:0) 1 .

Original: If f is assumed to be the sigmoid activation function, a l is then biased toward a speciﬁc region in (0 ; 1) m , because each element of a l have a different mean value from Equation (1).
Revised: If f is assumed to be the sigmoid activation function, a l is then biased toward a speciﬁc region in (0 ; 1) m , because each element of a l have a different mean value from Equation ( 1).

Original: In the next layer, the variance of (cid:18) l +1 i will shrink compared to that of (cid:18) li , because a l is biased in (0 ; 1) m .
Revised: In the next layer, the variance of (cid:18) l +1 i will shrink compared to that of (cid:18) li , because a l is biased in (0 ; 1) m .

Original: The variance of a l +1 i shrinks as the variance of (cid:18) l +1 i shrinks from Equations ( 1) and (3).
Revised: The variance of a l +1 i shrinks as the variance of (cid:18) l +1 i shrinks from Equations (1) and (3).

Original: Repeating the operations through multiple layers, the variance of (cid:18) li and a li will shrink to small values.
Revised: Repeating the operations through multiple layers, the variance of (cid:18) li and a li will shrink to small values.

Original: 2. V ISUALIZING EFFECT OF ANGLE BIAS USING CIFAR-10 DATASET
Revised: 2. V ISUALIZING EFFECT OF ANGLE BIAS USING CIFAR-10 DATASET

Original: We illustrate the effect of angle bias in an MLP by using the CIFAR-10 dataset ( Krizhevsky &
Revised: We illustrate the effect of angle bias in an MLP by using the CIFAR-10 dataset (Krizhevsky &

Original: Hinton, 2009) that includes a set of 32 (cid:2) 32 color (RGB) images.
Revised: Hinton, 2009) that includes a set of 32 (cid:2) 32 color (RGB) images.

Original: Each sample in CIFAR-10 is considered an input vector with 32 (cid:2) 32 (cid:2) 3 = 3072 real values, in which each variable is scaled into the range [ (cid:0) 1 ; 1] .
Revised: Each sample in CIFAR-10 is considered an input vector with 32 (cid:2) 32 (cid:2) 3 = 3072 real values, in which each variable is scaled into the range [ (cid:0) 1 ; 1] .

Original: 2.2. W ITH SIGMOID ACTIVATION FUNCTIONS
Revised: 2.2. W ITH SIGMOID ACTIVATION FUNCTIONS

Original: We consider an MLP with sigmoid activation functions that has 10 hidden layers with m =neurons in each layer.
Revised: We consider an MLP with sigmoid activation functions that has 10 hidden layers with m =neurons in each layer.

Original: The weights of the MLP are initialized according to Glorot & Bengio (2010).
Revised: The weights of the MLP are initialized according to Glorot & Bengio (2010).

Original: We randomly took 100 samples from the dataset and input them into the MLP.
Revised: We randomly took 100 samples from the dataset and input them into the MLP.

Original: Figure 2 shows the activation pattern in layers 1, 3, 5, 7, and 9 on the selected samples.
Revised: Figure 2 shows the activation pattern in layers 1, 3, 5, 7, and 9 on the selected samples.

Original: Please note that the activation in
Revised: Please note that the activation in

Original: Layer 1 corresponds to a 1 i in Equation (1), that is, Layer 1 is the layer after the input layer.
Revised: Layer 1 corresponds to a 1 i in Equation (1), that is, Layer 1 is the layer after the input layer.

Original: We see stripe patterns in the layers other than Layers 1 in Figure 2 that are caused by angle bias.
Revised: We see stripe patterns in the layers other than Layers 1 in Figure 2 that are caused by angle bias.

Original: In Layer 9, the activation value of each neuron is almost constant regardless of the input.
Revised: In Layer 9, the activation value of each neuron is almost constant regardless of the input.

Original: In contrast, no stripe pattern appears in Layer 1, because each element of the input vector is scaled into the range [ (cid:0) 1 ; 1] and its mean value is near zero; this corresponds to the case in which k ^ (cid:22) k (cid:25) 0 in Proposition 2.
Revised: In contrast, no stripe pattern appears in Layer 1, because each element of the input vector is scaled into the range [ (cid:0) 1 ; 1] and its mean value is near zero; this corresponds to the case in which k ^ (cid:22) k (cid:25) 0 in Proposition 2.

Original: We calculated the angle between w li and a l (cid:0) 1 , that is, (cid:18) li in Equation (3), for each sample 1 .
Revised: We calculated the angle between w li and a l (cid:0) 1 , that is, (cid:18) li in Equation (3), for each sample 1 .

Original: Figureshows boxplot summaries of (cid:18) li on the ﬁrst ten neurons in layers 1, 3, 5, 7, and 9, in which the 1%, 25%, 50%, 
Revised: Figureshows boxplot summaries of (cid:18) li on the ﬁrst ten neurons in layers 1, 3, 5, 7, and 9, in which the 1%, 25%, 50%, 75%, and 99% quantiles are displayed as whiskers or boxes. We see the mean of (cid:18) li are biased according to the neurons in the layers other than Layer 1. We also see that the variance of (cid:18) li shrink through layers.

Original: 2.2. W ITH R E LU ACTIVATION FUNCTIONS
Revised: 2.2. W ITH R E LU ACTIVATION FUNCTIONS

Original: Next, we consider an MLP with ReLU activation functions that has 50 hidden layers with m =neurons in each layer.
Revised: Next, we consider an MLP with ReLU activation functions that has 50 hidden layers with m =neurons in each layer.

Original: The weights are initialized according to Glorot & Bengio (2010).
Revised: The weights are initialized according to Glorot & Bengio (2010).

Original: Figureshows the activation pattern in layers 1, 10, 20, 30, and 40 on the randomly selected samples.
Revised: Figureshows the activation pattern in layers 1, 10, 20, 30, and 40 on the randomly selected samples.

Original: We see stripe patterns in the layers other than Layer 1 that are caused by the angle bias.
Revised: We see stripe patterns in the layers other than Layer 1 that are caused by the angle bias.

Original: Figure 5 shows boxplot summaries of (cid:18) li on the ﬁrst ten neurons in layers 1, 10, 20, 30, and 40.
Revised: Figure 5 shows boxplot summaries of (cid:18) li on the ﬁrst ten neurons in layers 1, 10, 20, 30, and 40.

Original: We see that the mean of (cid:18) li are biased according the neurons in the layers other than Layer 1.
Revised: We see that the mean of (cid:18) li are biased according the neurons in the layers other than Layer 1.

Original: We also see that the variance of (cid:18) li shrink through layers, but the shrinking rate is much moderate compared to that in Figure 3.
Revised: We also see that the variance of (cid:18) li shrink through layers, but the shrinking rate is much moderate compared to that in Figure 3.

Original: This is because ReLU projects a preactivation vector into the unbounded region [0 ; + 1 ) m and the activation vector is less likely to concentrate on a speciﬁc region.
Revised: This is because ReLU projects a preactivation vector into the unbounded region [0 ; + 1 ) m and the activation vector is less likely to concentrate on a speciﬁc region.

Original: 2. R ELATION TO VANISHING GRADIENT PROBLEM
Revised: 2. R ELATION TO VANISHING GRADIENT PROBLEM

Original: Under the effect of angle bias, the activation of neurons in deeper layers are almost constant regardless of the input in an MLP with sigmoid activation functions, as shown in Figure 2.
Revised: Under the effect of angle bias, the activation of neurons in deeper layers are almost constant regardless of the input in an MLP with sigmoid activation functions, as shown in Figure 2.

Original: It indicates that ∇ a 0 L = 0 , where L is a loss function that is deﬁned based on the output of the MLP and ∇ a 0 L means the gradient with respect to the input vector a 0 .
Revised: It indicates that ∇ a 0 L = 0 , where L is a loss function that is deﬁned based on the output of the MLP and ∇ a 0 L means the gradient with respect to the input vector a 0 .

Original: From Equation (2), we have
Revised: From Equation (2), we have

Original: [Equation]
Revised: [Equation]

Original: Assuming that w 1 i ( i = 1 ; : : : ; m ) are linearly independent, it follows that ∇ z 1 i = 

Revised: Assuming that w 1 i ( i = 1 ; : : : ; m ) are linearly independent, it follows that ∇ z 1 i =

Original: ( i = 1 ; : : : ; m ) from Equation (4), with l = 1 , and ∇ a 0 L = 0 .
Revised: ( i = 1 ; : : : ; m ) from Equation (4), with l = 1 , and ∇ a 0 L = 0 .

Original: Then, it holds that ∇ w 1 i L = 

Revised: Then, it holds that ∇ w 1 i L =

Original: ( i = 1 ; : : : ; m ) from
Revised: ( i = 1 ; : : : ; m ) from

Original: Equation (5), with l = 1 , indicating that the gradients of weights in the ﬁrst layer are vanished.
Revised: Equation (5), with l = 1 , indicating that the gradients of weights in the ﬁrst layer are vanished.

Original: From
Revised: From

Original: Equation (1), with l = 1 , we have ∇ z 1 i L = f 0 ( z 1 i ) ∇ a 1 i L .
Revised: Equation (1), with l = 1 , we have ∇ z 1 i L = f 0 ( z 1 i ) ∇ a 1 i L .

Original: If f 0 ( z 1 i ) 6 = 0 , it follows that ∇ a 1 i L =from ∇ z 1 i L = 0 .
Revised: If f 0 ( z 1 i ) 6 = 0 , it follows that ∇ a 1 i L =from ∇ z 1 i L = 0 .

Original: This leads to ∇ w 2 i L = 0 from Equations (4) and (5), with l = 2 , under the assumption that w 2 i ( i = 1 ; : : : ; m ) are linearly independent.
Revised: This leads to ∇ w 2 i L = 0 from Equations (4) and (5), with l = 2 , under the assumption that w 2 i ( i = 1 ; : : : ; m ) are linearly independent.

Original: Consequently, we can derive ∇ w li L = 0 from ∇ a 0 L = 

Revised: Consequently, we can derive ∇ w li L = 0 from ∇ a 0 L =

Original: so far as f 0 ( z li ) 6 = 0 and w li ( i = 1 ; : : : ; m ) being linearly independent.
Revised: so far as f 0 ( z li ) 6 = 0 and w li ( i = 1 ; : : : ; m ) being linearly independent.

Original: If we use rectiﬁed linear activation instead of sigmoid activation, the gradients of weights are less likely to vanish, because ∇ a 0 L will seldom be exactly zero.
Revised: If we use rectiﬁed linear activation instead of sigmoid activation, the gradients of weights are less likely to vanish, because ∇ a 0 L will seldom be exactly zero.

Original: However, the rate of each neuron being active 2 is biased, because the distribution of preactivation z li is biased.
Revised: However, the rate of each neuron being active 2 is biased, because the distribution of preactivation z li is biased.

Original: If a neuron is always active, it behaves as an identity mapping.
Revised: If a neuron is always active, it behaves as an identity mapping.

Original: If a neuron is always inactive, it is worthless because its output is always zero.
Revised: If a neuron is always inactive, it is worthless because its output is always zero.

Original: Such a phenomenon is observed in deep layers in Figure 4.
Revised: Such a phenomenon is observed in deep layers in Figure 4.

Original: As discussed in Balduzzi et al.
Revised: As discussed in Balduzzi et al.

Original: (2017), the efﬁciency of the network decreases in this case.
Revised: (2017), the efﬁciency of the network decreases in this case.

Original: In this sense, angle bias may reduce the efﬁciency of a network with rectiﬁed linear activation.
Revised: In this sense, angle bias may reduce the efﬁciency of a network with rectiﬁed linear activation.

Original:  L INEARLY CONSTRAINED WEIGHTS
Revised:  L INEARLY CONSTRAINED WEIGHTS

Original: There are two approaches to reduce angle bias in a neural network.
Revised: There are two approaches to reduce angle bias in a neural network.

Original: The ﬁrst one is to somehow make the expected value of the activation of each neuron near zero, because angle bias does not occur if k ^ (cid:22) k
Revised: The ﬁrst one is to somehow make the expected value of the activation of each neuron near zero, because angle bias does not occur if k ^ (cid:22) k

Original: = 0 from Proposition 2.
Revised: = 0 from Proposition 2.

Original: The second one is to somehow regularize the angle between w li and E ( a l (cid:0) 1 ) .
Revised: The second one is to somehow regularize the angle between w li and E ( a l (cid:0) 1 ) .

Original: In this section, we propose a method to reduce angle bias in a neural network by using the latter approach.
Revised: In this section, we propose a method to reduce angle bias in a neural network by using the latter approach.

Original: We introduce W LC as follows:
Revised: We introduce W LC as follows:

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: W LC is a subspace in R m deﬁned by
Revised: W LC is a subspace in R m deﬁned by

Original: [Equation]
Revised: [Equation]

Original: The following holds for w 2 W LC :
Revised: The following holds for w 2 W LC :

Original: Proposition 3.
Revised: Proposition 3.

Original: Let a be an m dimensional random variable that follows P (cid:13) .
Revised: Let a be an m dimensional random variable that follows P (cid:13) .

Original: Given w 2 W LC such that k w k > 0 , the expected value of w (cid:1) a is zero.
Revised: Given w 2 W LC such that k w k > 0 , the expected value of w (cid:1) a is zero.

Original: Proof.
Revised: Proof.

Original: E ( w (cid:1)
Revised: E ( w [Equation] =

Original: [Equation]
Revised: [Equation]

Revised: E (

Original: : : : + w m ) = 0 .
Revised: [Equation] = (cid:13) [Equation] = 0 .

Original: From Proposition 3, if w li 2 W LC and a l (cid:0) 1 follows P (cid:13) , we can resolve angle bias in z li in Equation (2).
Revised: From Proposition 3, if w li 2 W LC and a l (cid:0) 1 follows P (cid:13) , we can resolve angle bias in z li in Equation (2).

Original: If we initialize b li = 0 , the distribution of each of z li ( i = 1 ; : : : ; m ) will likely be more similar to each other.
Revised: If we initialize b li = 0 , the distribution of each of z li ( i = 1 ; : : : ; m ) will likely be more similar to each other.

Original: The activation vector in layer l , each of whose elements is given by Equation (1), is then expected to follow P (cid:13) .
Revised: The activation vector in layer l , each of whose elements is given by Equation (1), is then expected to follow P (cid:13) .

Original: Therefore, if the input vector a 0 follows P (cid:13) , we can inductively reduce the angle bias in each layer of an MLP by using weight vectors that are included in W LC .
Revised: Therefore, if the input vector a 0 follows P (cid:13) , we can inductively reduce the angle bias in each layer of an MLP by using weight vectors that are included in W LC .

Original: We call weight vector w li in W LC linearly constrained weights (LCWs) .
Revised: We call weight vector w li in W LC linearly constrained weights (LCWs) .

Original: 3. V ISUALIZING THE EFFECT OF LCW 3.1. W ITH SIGMOID ACTIVATION FUNCTIONS
Revised: 3. V ISUALIZING THE EFFECT OF LCW 3.1. W ITH SIGMOID ACTIVATION FUNCTIONS

Original: We built an MLP with sigmoid activation functions of the same size as that used in Section 2.2.1, but whose weight vectors are replaced with LCWs.
Revised: We built an MLP with sigmoid activation functions of the same size as that used in Section 2.2.1, but whose weight vectors are replaced with LCWs.

Original: We applied the minibatch-based initialization described in Section 3.3.
Revised: We applied the minibatch-based initialization described in Section 3.3.

Original: Figure 6 shows the activation pattern in layers 1, 3, 5, 7, and 9 of the
Revised: Figure 6 shows the activation pattern in layers 1, 3, 5, 7, and 9 of the

Original: MLP with LCW on the randomly selected samples that are used in Figure 2.
Revised: MLP with LCW on the randomly selected samples that are used in Figure 2.

Original: When compared with
Revised: When compared with

Original: Figure 2, we see no stripe pattern in Figure 6.
Revised: Figure 2, we see no stripe pattern in Figure 6.

Original: The neurons in Layer 9 respond differently to each input sample; this means that a change in the input leads to a different output.
Revised: The neurons in Layer 9 respond differently to each input sample; this means that a change in the input leads to a different output.

Original: Therefore, the network output changes if we adjust the weight vectors in Layer 1, that is, the gradients of weights in Layerdo not vanish in Figure 6.
Revised: Therefore, the network output changes if we adjust the weight vectors in Layer 1, that is, the gradients of weights in Layerdo not vanish in Figure 6.

Original: Figure 7 shows boxplot summaries of (cid:18) li on the ﬁrst ten neurons in layers 1, 3, 5, 7, and 9 of the
Revised: Figure 7 shows boxplot summaries of (cid:18) li on the ﬁrst ten neurons in layers 1, 3, 5, 7, and 9 of the

Original: MLP with LCW.
Revised: MLP with LCW.

Original: We see that the angle distributes around 90 ◦ on each neuron in each layer.
Revised: We see that the angle distributes around 90 ◦ on each neuron in each layer.

Original: This indicates that the angle bias is resolved in the calculation of z li by using LCW.
Revised: This indicates that the angle bias is resolved in the calculation of z li by using LCW.

Original: After 10 epochs training Figure 8 shows the activation pattern in layers of the MLP with LCW after 10 epochs training.
Revised: After 10 epochs training Figure 8 shows the activation pattern in layers of the MLP with LCW after 10 epochs training.

Original: A slight stripe pattern is visible in Figure 8, but neurons in each layer react differently to each input.
Revised: A slight stripe pattern is visible in Figure 8, but neurons in each layer react differently to each input.

Original: Figure 9 shows boxplot summaries of (cid:18) li of the MLP after 10 epochs training.
Revised: Figure 9 shows boxplot summaries of (cid:18) li of the MLP after 10 epochs training.

Original: We see that the mean of (cid:18) li is slightly biased according to the neurons.
Revised: We see that the mean of (cid:18) li is slightly biased according to the neurons.

Original: However, the variance of (cid:18) li do not shrink even in deeper layers.
Revised: However, the variance of (cid:18) li do not shrink even in deeper layers.

Original: 3.1. W ITH R E LU ACTIVATION FUNCTIONS
Revised: 3.1. W ITH R E LU ACTIVATION FUNCTIONS

Original: We built an MLP with ReLU activation functions of the same size as that used in Section 2.2.2, whose weight vectors are replaced with LCWs.
Revised: We built an MLP with ReLU activation functions of the same size as that used in Section 2.2.2, whose weight vectors are replaced with LCWs.

Original: We applied the minibatch-based initialization described in Section 3.3.
Revised: We applied the minibatch-based initialization described in Section 3.3.

Original: Figure 10 shows the activation pattern in layers 1, 10, 20, 30, and 40 of the
Revised: Figure 10 shows the activation pattern in layers 1, 10, 20, 30, and 40 of the

Original: MLP with LCW.
Revised: MLP with LCW.

Original: When compared with Figure 4, we see no stripe pattern in Figure 10.
Revised: When compared with Figure 4, we see no stripe pattern in Figure 10.

Original: Figureshows boxplot summaries of (cid:18) l i on the ﬁrst ten neurons in layers 1, 10, 20, 30, and 40 of the MLP with LCW.
Revised: Figureshows boxplot summaries of (cid:18) li on the ﬁrst ten neurons in layers 1, 10, 20, 30, and 40 of the MLP with LCW.

Original: We can observe that the angle bias is resolved by using LCW in the MLP with ReLU activation functions.
Revised: We can observe that the angle bias is resolved by using LCW in the MLP with ReLU activation functions.

Original: 3. L EARNING LCW VIA REPARAMETERIZATION
Revised: 3. L EARNING LCW VIA REPARAMETERIZATION

Original: A straightforward way to train a neural network with LCW is to solve a constrained optimization problem, in which a loss function is minimized under the condition that each weight vector is included in W LC .
Revised: A straightforward way to train a neural network with LCW is to solve a constrained optimization problem, in which a loss function is minimized under the condition that each weight vector is included in W LC .

Original: Although several methods are available to solve such constrained problems, for example, the gradient projection method (Luenberger & Ye, 2015), it might be less efﬁcient to solve a constrained optimization problem than to solve an unconstrained one.
Revised: Although several methods are available to solve such constrained problems, for example, the gradient projection method ( Luenberger & Ye, 2015), it might be less efﬁcient to solve a constrained optimization problem than to solve an unconstrained one.

Original: We propose a reparameterization technique that enables us to train a neural network with LCW by using a solver for unconstrained optimization.
Revised: We propose a reparameterization technique that enables us to train a neural network with LCW by using a solver for unconstrained optimization.

Original: We can embed the constraints on the weight vectors into the structure of the neural network by reparameterization.
Revised: We can embed the constraints on the weight vectors into the structure of the neural network by reparameterization.

Original: Reparameterization Let w li 2 R m be a weight vector in a neural network.
Revised: Reparameterization Let w li 2 R m be a weight vector in a neural network.

Original: We reparametrize w li by using vector v l i 2 R m (cid:0) 1 as w l i =
Revised: We reparametrize w li by using vector v li 2 R m (cid:0) 1 as w li =

Original: B m v l i , where B m 2 R m (cid:2) ( m (cid:0) 1) is a basis of W LC , written as a matrix of column vectors.
Revised: B m v li , where B m 2 R m (cid:2) ( m (cid:0) 1) is a basis of W LC , written as a matrix of column vectors.

Original: For example, B m is given by
Revised: For example, B m is given by

Original: [Equation]
Revised: [Equation]

Original: where I m (cid:0) 1 is the identity matrix of order ( m (cid:0) 1) (cid:2) ( m (cid:0) 1) .
Revised: where I m (cid:0) 1 is the identity matrix of order ( m (cid:0) 1) (cid:2) ( m (cid:0) 1) .

Original: It is obvious that w li =
Revised: It is obvious that w li =

Original: B m v li 2 W LC .
Revised: B m v li 2 W LC .

Original: We then solve the optimization problem in which v li is considered as a new variable in place of w li .
Revised: We then solve the optimization problem in which v li is considered as a new variable in place of w li .

Original: This optimization problem is unconstrained because v li 2 R m (cid:0) 1 .
Revised: This optimization problem is unconstrained because v li 2 R m (cid:0) 1 .

Original: We can search for w li 2 W LC by exploring v li 2 R m (cid:0) 1 .
Revised: We can search for w li 2 W LC by exploring v li 2 R m (cid:0) 1 .

Original: In the experiments in
Revised: In the experiments in

Original: Section 5, we used B m in Equation (7).
Revised: Section 5, we used B m in Equation (7).

Original: We also tried an orthonormal basis of W LC as B m ; however, there was little difference in accuracy.
Revised: We also tried an orthonormal basis of W LC as B m ; however, there was little difference in accuracy.

Original: It is worth noting that the proposed reparameterization can be implemented easily and efﬁciently by using modern frameworks for deep learning using GPUs.
Revised: It is worth noting that the proposed reparameterization can be implemented easily and efﬁciently by using modern frameworks for deep learning using GPUs.

Original: 3. I NITIALIZATION USING MINIBATCH STATISTICS
Revised: 3. I NITIALIZATION USING MINIBATCH STATISTICS

Original: By introducing LCW, we can reduce the angle bias in z li in Equation (2), which mainly affects the expected value of z li .
Revised: By introducing LCW, we can reduce the angle bias in z l i in Equation (2), which mainly affects the expected value of z li .

Original: It is also important to regularize the variance of z li , especially when the sigmoid activation is used, because the output of the activation will likely saturate when the variance of z l i is too large.
Revised: It is also important to regularize the variance of z li , especially when the sigmoid activation is used, because the output of the activation will likely saturate when the variance of z li is too large.

Original: We apply an initialization method by which the variance of z li is regularized based on a minibatch of samples.
Revised: We apply an initialization method by which the variance of z li is regularized based on a minibatch of samples.

Original: This type of initialization has also been used in previous studies Mishkin &
Revised: This type of initialization has also been used in previous studies Mishkin &

Original: Matas (2016) and Salimans & Kingma (2016).
Revised: Matas (2016) and Salimans & Kingma (2016).

Original: First, we randomly generate a seed vector ^ w li [Equation] W LC1 , where W LC
Revised: First, we randomly generate a seed vector ^ w li 2 W LC1 , where W LC

Original: [Equation]
Revised: [Equation]

Original: ={
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Section A in the appendix describes the approach for sampling uniformly from W LC1 .
Revised: Section A in the appendix describes the approach for sampling uniformly from W LC1 .

Original: The scholar variable (cid:17) l is then calculated in a layer-wise manner with the property that the standard deviation of ( (cid:17) l ^ w li ) (cid:1) a l (cid:0) 1 equals the target value (cid:27) z , in which a l (cid:0) 1 is evaluated using a minibatch.
Revised: The scholar variable (cid:17) l is then calculated in a layer-wise manner with the property that the standard deviation of ( (cid:17) l ^ w li ) (cid:1) a l (cid:0) 1 equals the target value (cid:27) z , in which a l (cid:0) 1 is evaluated using a minibatch.

Original: Finally, we use (cid:17) l ^ w li as the initial value of w li .
Revised: Finally, we use (cid:17) l ^ w li as the initial value of w li .

Original: If we apply the reparameterization method described in the previous section, the ﬁrst m (cid:0) 1 elements of (cid:17) l ^ w li are used as the initial value of v li .
Revised: If we apply the reparameterization method described in the previous section, the ﬁrst m (cid:0) 1 elements of (cid:17) l ^ w li are used as the initial value of v li .

Original: We initialize bias terms b li in Equation (2) to zero.
Revised: We initialize bias terms b li in Equation (2) to zero.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Ioffe & Szegedy (2015) proposed the batch normalization (BN) approach for accelerating the training of deep nets.
Revised: Ioffe & Szegedy (2015) proposed the batch normalization (BN) approach for accelerating the training of deep nets.

Original: BN was developed to address the problem of internal covariate shift , that is, training deep nets is difﬁcult because the distribution of the input of a layer changes as the weights of the preceding layers change during training.
Revised: BN was developed to address the problem of internal covariate shift , that is, training deep nets is difﬁcult because the distribution of the input of a layer changes as the weights of the preceding layers change during training.

Original: The computation of the mean and standard deviation of z li based on a minibatch is incorporated into the structure of the network, and z li is normalized by using these statistics.
Revised: The computation of the mean and standard deviation of z li based on a minibatch is incorporated into the structure of the network, and z li is normalized by using these statistics.

Original: G ¨ ulc¸ehre & Bengio (2016) proposed the standardization layer (SL) approach, which is similar to BN.
Revised: G¨ulc¸ehre & Bengio (2016) proposed the standardization layer (SL) approach, which is similar to BN.

Original: The main difference is that SL normalizes a li , whereas BN normalizes z li .
Revised: The main difference is that SL normalizes a li , whereas BN normalizes z li .

Original: Interestingly, both BN and SL can be considered mechanisms for reducing the angle bias.
Revised: Interestingly, both BN and SL can be considered mechanisms for reducing the angle bias.

Original: SL reduces the angle bias by forcing k ^ (cid:22) k
Revised: SL reduces the angle bias by forcing k ^ (cid:22) k

Original: = 0 in Proposition 2.
Revised: = 0 in Proposition 2.

Original: On the other hand, BN reduces the angle bias by normalizing z li for each neuron.
Revised: On the other hand, BN reduces the angle bias by normalizing z li for each neuron.

Original: A drawback of both BN and SL is that the model has to be switched during inference to ensure that its output depends only on the input but not the minibatch.
Revised: A drawback of both BN and SL is that the model has to be switched during inference to ensure that its output depends only on the input but not the minibatch.

Original: By contrast, LCW proposed in this paper does not require any change in the model during inference.
Revised: By contrast, LCW proposed in this paper does not require any change in the model during inference.

Original: Moreover, the computational overhead for training the LCW is lower than that of BN and SL, because LCW only requires the addition of layers that multiply B m to v li , as described in Section 3.2, whereas BN and SL need to compute both the mean and the standard deviation of z li or a li .
Revised: Moreover, the computational overhead for training the LCW is lower than that of BN and SL, because LCW only requires the addition of layers that multiply B m to v li , as described in Section 3.2, whereas BN and SL need to compute both the mean and the standard deviation of z li or a li .

Original: Salimans & Kingma (2016) proposed weight normalization (WN) to overcome the drawbacks of
Revised: Salimans & Kingma (2016) proposed weight normalization (WN) to overcome the drawbacks of

Original: BN, in which a weight vector w li 2 R m is reparametrized as w li = 
Revised: BN, in which a weight vector w l i 2 R m is reparametrized as w l i = ( g l i = k

Original: ( g li = k v li k ) v li , where g li 2 R and v li 2 R m are new parameters.
Revised: v l i k ) v l i , where g l i 2 R and v li 2 R m are new parameters.

Original: By deﬁnition, WN does not have the property of reducing the angle bias, because the degrees of freedom of w l i are unchanged by the reparameterization.
Revised: By deﬁnition, WN does not have the property of reducing the angle bias, because the degrees of freedom of w li are unchanged by the reparameterization.

Original: Salimans & Kingma (2016) also proposed a minibatch-based initialization by which weight vectors are initialized so that z li has zero mean and unit variance, indicating that the angle bias is reduced immediately after the initialization.
Revised: Salimans & Kingma (2016) also proposed a minibatch-based initialization by which weight vectors are initialized so that z li has zero mean and unit variance, indicating that the angle bias is reduced immediately after the initialization.

Original: Ba et al.
Revised: Ba et al.

Original: (2016) proposed layer normalization (LN) as a variant of BN.
Revised: (2016) proposed layer normalization (LN) as a variant of BN.

Original: LN normalizes z li over the neurons in a layer on a sample in the minibatch, whereas BN normalizes z li over the minibatch on a neuron.
Revised: LN normalizes z li over the neurons in a layer on a sample in the minibatch, whereas BN normalizes z li over the minibatch on a neuron.

Original: From the viewpoint of reducing the angle bias, LN is not as direct as BN.
Revised: From the viewpoint of reducing the angle bias, LN is not as direct as BN.

Original: Although LN does not resolve the angle bias, it is expected to normalize the degree of bias in each layer.
Revised: Although LN does not resolve the angle bias, it is expected to normalize the degree of bias in each layer.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conducted preliminary experiments using the CIFAR-10 dataset, the CIFARdataset (Krizhevsky & Hinton, 2009), and the SVHN dataset (Netzer et al., 2011).
Revised: We conducted preliminary experiments using the CIFAR-10 dataset, the CIFARdataset (Krizhevsky & Hinton, 2009), and the SVHN dataset (Netzer et al., 2011).

Original: These experiments are aimed not at achieving state-of-the-art results but at investigating whether we can train a deep model by reducing the angle bias and empirically evaluating the performance of LCW in comparison to that of BN and WN.
Revised: These experiments are aimed not at achieving state-of-the-art results but at investigating whether we can train a deep model by reducing the angle bias and empirically evaluating the performance of LCW in comparison to that of BN and WN.

Original: Network structure We used MLPs with the cross-entropy loss function.
Revised: Network structure We used MLPs with the cross-entropy loss function.

Original: Each network has 32 (cid:2) 32 (cid:2) 3 = 3072 input neurons and 10 output neurons, and it is followed by a softmax layer.
Revised: Each network has 32 (cid:2) 32 (cid:2) 3 = 3072 input neurons and 10 output neurons, and it is followed by a softmax layer.

Original: We refer to an MLP that has L hidden layers and M neurons in each hidden layer as MLP ( L; M ) .
Revised: We refer to an MLP that has L hidden layers and M neurons in each hidden layer as MLP ( L; M ) .

Original: Either a sigmoid activation function or a rectiﬁed linear activation function was used.
Revised: Either a sigmoid activation function or a rectiﬁed linear activation function was used.

Original: MLP LCW denotes an MLP in which each weight vector is replaced by LCW.
Revised: MLP LCW denotes an MLP in which each weight vector is replaced by LCW.

Original: MLP BN denotes an MLP in which the preactivation of each neuron is normalized by BN.
Revised: MLP BN denotes an MLP in which the preactivation of each neuron is normalized by BN.

Original: MLP WN denotes an MLP whose weight vectors are reparametrized by WN.
Revised: MLP WN denotes an MLP whose weight vectors are reparametrized by WN.

Original: Initialization Plain MLP and MLP BN were initialized using the method proposed in Glorot &
Revised: Initialization Plain MLP and MLP BN were initialized using the method proposed in Glorot &

Original: Bengio (2010).
Revised: Bengio (2010).

Original: MLP LCW was initialized using the minibatch-based method described in Section 3. with (cid:27)
Revised: MLP LCW was initialized using the minibatch-based method described in Section 3. with (cid:27)

Original: z
Revised: z

Original: = 0 : 5 .
Revised: = 0 : 5 .

Original: MLP WN was initialized according to Salimans & Kingma (2016).
Revised: MLP WN was initialized according to Salimans & Kingma (2016).

Original: Optimization MLPs were trained using a stochastic gradient descent with a minibatch size offor 100 epochs.
Revised: Optimization MLPs were trained using a stochastic gradient descent with a minibatch size offor 100 epochs.

Original: The learning rate starts from 0 : 1 and is multiplied by 0 : 95 after every two epochs.
Revised: The learning rate starts from 0 : 1 and is multiplied by 0 : 95 after every two epochs.

Original: Environment and implementation The experiments were performed on a system running Ubuntu 16.04 LTS with NVIDIA R (cid:13) Tesla R (cid:13) K80 GPUs.
Revised: Environment and implementation The experiments were performed on a system running Ubuntu 16.04 LTS with NVIDIA R (cid:13) Tesla R (cid:13) K80 GPUs.

Original: We implemented LCW using PyTorch version 0.1.12.
Revised: We implemented LCW using PyTorch version 0.1.12.

Original: We implemented BN using the torch.nn.
Revised: We implemented BN using the torch.nn.

Original: BatchNorm1d module in PyTorch.
Revised: BatchNorm1d module in PyTorch.

Original: We implemented WN by ourselves using PyTorch 3 .
Revised: We implemented WN by ourselves using PyTorch 3 .

Original: 5. E XPERIMENTAL RESULTS : MLP S WITH SIGMOID ACTIVATION FUNCTIONS
Revised: 5. E XPERIMENTAL RESULTS : MLP S WITH SIGMOID ACTIVATION FUNCTIONS

Original: We ﬁrst consider MLPs with sigmoid activation functions.
Revised: We ﬁrst consider MLPs with sigmoid activation functions.

Original: Figure 12 shows the convergence and computation time for training MLPs with CIFAR-10 dataset.
Revised: Figure 12 shows the convergence and computation time for training MLPs with CIFAR-10 dataset.

Original: Figure 12(a)-(c) shows results for MLP (100 ; 128) , Figure 12(d)-(f) for MLP (50 ; 256) , and Figure 12(g)-(i) for MLP (5 ; 512) .
Revised: Figure 12(a)-(c) shows results for MLP (100 ; 128) , Figure 12(d)-(f) for MLP (50 ; 256) , and Figure 12(g)-(i) for MLP (5 ; 512) .

Original: Figure 12(a) shows that the training accuracy of the plain MLP (100 ; 128) is 10% throughout the training, because the MLP output is insensible to the input because of the angle bias, as mentioned in
Revised: Figure 12(a) shows that the training accuracy of the plain MLP (100 ; 128) is 10% throughout the training, because the MLP output is insensible to the input because of the angle bias, as mentioned in

Original: Section 2.2 4 .
Revised: Section 2.2 4 .

Original: By contrast, MLP LCW or MLP BN is successfully trained, as shown in Figure 12(a), indicating that the angle bias is a crucial obstacle to training deep MLPs with sigmoid activation functions.
Revised: By contrast, MLP LCW or MLP BN is successfully trained, as shown in Figure 12(a), indicating that the angle bias is a crucial obstacle to training deep MLPs with sigmoid activation functions.

Original: MLP LCW achieves a higher rate of increase in the training accuracy compared to MLP BN in Figure 12(a), (d), and (g).
Revised: MLP LCW achieves a higher rate of increase in the training accuracy compared to MLP BN in Figure 12(a), (d), and (g).

Original: As described in Section 4, WN itself cannot reduce the angle bias, but the bias is reduced immediately after the initialization of WN.
Revised: As described in Section 4, WN itself cannot reduce the angle bias, but the bias is reduced immediately after the initialization of WN.

Original: From Figure 12(a) and (d), we see that deep MLPs with WN are not trainable.
Revised: From Figure 12(a) and (d), we see that deep MLPs with WN are not trainable.

Original: These results suggest that starting with weight vectors that do not incur angle bias is not sufﬁcient to train deep nets.
Revised: These results suggest that starting with weight vectors that do not incur angle bias is not sufﬁcient to train deep nets.

Original: It is important to incorporate a mechanism that reduces the angle bias during training, such as LCW or BN.
Revised: It is important to incorporate a mechanism that reduces the angle bias during training, such as LCW or BN.

Original: The computational overhead of training of MLP LCW (100 ; 128) is approximately 55% compared to plain MLP (100 ; 128) , as shown in Figure 12(b); this is much lower than that of MLP BN (100 ; 128) .
Revised: The computational overhead of training of MLP LCW (100 ; 128) is approximately 55% compared to plain MLP (100 ; 128) , as shown in Figure 12(b); this is much lower than that of MLP BN (100 ; 128) .

Original: The overhead of MLP WN is large compared to that of MLP BN , although it contradicts the claim of
Revised: The overhead of MLP WN is large compared to that of MLP BN , although it contradicts the claim of

Original: Salimans & Kingma (2016).
Revised: Salimans & Kingma (2016).

Original: We think this is due to the implementation of these methods.
Revised: We think this is due to the implementation of these methods.

Original: The BN  module we used in the experiments consists of a specialized function developed by GPU vendors, whereas the WN module was developed by ourselves.
Revised: The BN [Figure] module we used in the experiments consists of a specialized function developed by GPU vendors, whereas the WN module was developed by ourselves.

Original: In this sense, the overhead of LCW may be improved by a more sophisticated implementation.
Revised: In this sense, the overhead of LCW may be improved by a more sophisticated implementation.

Original: In terms of the test accuracy, MLP LCW has peaks around 20 epochs, as shown in Figure 12(c), (f), and (i).
Revised: In terms of the test accuracy, MLP LCW has peaks around 20 epochs, as shown in Figure 12(c), (f), and (i).

Original: We have no clear explanation for this ﬁnding, and further studies are needed to investigate the generalizability of neural networks.
Revised: We have no clear explanation for this ﬁnding, and further studies are needed to investigate the generalizability of neural networks.

Original: Experimental results with SVHN and CIFAR-100 datasets are reported in Section B in the appendix.
Revised: Experimental results with SVHN and CIFAR-100 datasets are reported in Section B in the appendix.

Original: 5.  E XPERIMENTAL RESULTS : MLP S WITH RECTIFIED LINEAR ACTIVATION FUNCTIONS
Revised:  E XPERIMENTAL RESULTS : MLP S WITH RECTIFIED LINEAR ACTIVATION FUNCTIONS

Original: We have experimented with MLPs with rectiﬁed linear activation functions.
Revised: We have experimented with MLPs with rectiﬁed linear activation functions.

Original: In our experiments, we observed that the plain MLP with 20 layers and 256 neurons per layer was successfully trained.
Revised: In our experiments, we observed that the plain MLP with 20 layers and 256 neurons per layer was successfully trained.

Original: However, the training of MLP LCW of the same size did not proceed at all, regardless of the dataset[Figure] used in our experiment; in fact, the output values of the network exploded after a few minibatch updates.
Revised: However, the training of MLP LCW of the same size did not proceed at all, regardless of the dataset used in our experiment; in fact, the output values of the network exploded after a few minibatch updates.

Original: We have investigated the weight gradients of the plain MLP and MLP LCW .
Revised: We have investigated the weight gradients of the plain MLP and MLP LCW .

Original: Figure 13 shows boxplot summaries of the weight gradients in each layer of both models, in which the gradients are evaluated by using a minibatch of CIFAR-10 immediately after the initialization.
Revised: Figure 13 shows boxplot summaries of the weight gradients in each layer of both models, in which the gradients are evaluated by using a minibatch of CIFAR-10 immediately after the initialization.

Original: By comparing
Revised: By comparing

Original: Figure 13(a) and Figure 13(b), we ﬁnd an exponential increase in the distributions of the weight gradients of MLP LCW in contrast to the plain MLP.
Revised: Figure 13(a) and Figure 13(b), we ﬁnd an exponential increase in the distributions of the weight gradients of MLP LCW in contrast to the plain MLP.

Original: Because the learning rate was the same for every layer in our experiments, this exponential increase of the gradients might hamper the learning of
Revised: Because the learning rate was the same for every layer in our experiments, this exponential increase of the gradients might hamper the learning of

Original: MLP LCW .
Revised: MLP LCW .

Original: The gradients in a rectiﬁer network are sums of path-weights over active paths (Balduzzi et al., 2017).
Revised: The gradients in a rectiﬁer network are sums of path-weights over active paths (Balduzzi et al., 2017).

Original: The exponential increase of the gradients therefore implies an exponential increase  of active paths.
Revised: The exponential increase of the gradients therefore implies an exponential increase [Figure] of active paths.

Original: As discussed in Section 2.3, we can prevent neurons from being always inactive by reducing the angle bias, which we think caused the exponential increase in active paths.
Revised: As discussed in Section 2.3, we can prevent neurons from being always inactive by reducing the angle bias, which we think caused the exponential increase in active paths.

Original: We need further studies to make MLP LCW with rectiﬁed linear activation functions trainable.
Revised: We need further studies to make MLP LCW with rectiﬁed linear activation functions trainable.

Original: Possible directions are to apply layer-wise learning rates or to somehow regularize the distribution of the weight gradients in each layer of MLP LCW , which we leave as future work.
Revised: Possible directions are to apply layer-wise learning rates or to somehow regularize the distribution of the weight gradients in each layer of MLP LCW , which we leave as future work.

Original:  C ONCLUSIONS AND FUTURE WORK
Revised:  C ONCLUSIONS AND FUTURE WORK

Original: In this paper, we have ﬁrst identiﬁed the angle bias that arises in the dot product of a nonzero vector and a random vector.
Revised: In this paper, we have ﬁrst identiﬁed the angle bias that arises in the dot product of a nonzero vector and a random vector.

Original: The mean of the dot product depends on the angle between the nonzero vector and the mean vector of the random vector.
Revised: The mean of the dot product depends on the angle between the nonzero vector and the mean vector of the random vector.

Original: In a neural network, the preactivation value of a neuron is biased depending on the angle between the weight vector of the neuron and the mean of the activation vector in the previous layer.
Revised: In a neural network, the preactivation value of a neuron is biased depending on the angle between the weight vector of the neuron and the mean of the activation vector in the previous layer.

Original: We have shown that such biases cause a vanishing gradient in a neural[Figure] network with sigmoid activation functions.
Revised: We have shown that such biases cause a vanishing gradient in a neural network with sigmoid activation functions.

Original: To overcome this problem, we have proposed linearly constrained weights to reduce the angle bias in a neural network; these can be learned efﬁciently by the reparameterization technique.
Revised: To overcome this problem, we have proposed linearly constrained weights to reduce the angle bias in a neural network; these can be learned efﬁciently by the reparameterization technique.

Original: Preliminary experiments suggest that reducing the angle bias is essential to train deep MLPs with sigmoid activation functions.
Revised: Preliminary experiments suggest that reducing the angle bias is essential to train deep MLPs with sigmoid activation functions.

Original: We have observed that reducing the angle bias causes an unfavorable side effect in the training of
Revised: We have observed that reducing the angle bias causes an unfavorable side effect in the training of

Original: MLPs with rectiﬁed linear activation functions, in which the gradient exploding problem occurs.
Revised: MLPs with rectiﬁed linear activation functions, in which the gradient exploding problem occurs.

Original: We need further studies to make LCW applicable to rectiﬁer networks.
Revised: We need further studies to make LCW applicable to rectiﬁer networks.

Original: Future work also includes investigating the applicability of LCW for other neural network structures, such as convolutional or recurrent structures.
Revised: Future work also includes investigating the applicability of LCW for other neural network structures, such as convolutional or recurrent structures.

Original: The use of LCW in recurrent networks is of particular interest, for which batch normalization is not straightforwardly applicable
Revised: The use of LCW in recurrent networks is of particular interest, for which batch normalization is not straightforwardly applicable.

Original: A S AMPLING UNIFORMLY FROM W LC 
Revised: A S AMPLING UNIFORMLY FROM W LC

Original: We can generate a sample w that is uniformly distributed in W LC1 as follows:
Revised: We can generate a sample w that is uniformly distributed in W LC1 as follows:

Original: w (cid:24) N (0 ; I m )
Revised: w (cid:24) N (0 ; I m )

Original: : Generate a sample w from m -dimensional normal distribution N (0 ; I m ) .
Revised: : Generate a sample w from m -dimensional normal distribution N (0 ; I m ) .

Original: w w (cid:0) E ( w ) : Subtract the mean of elements of w , which corresponds to the projection of w onto W LC . 3.
Revised: w w (cid:0) E ( w ) : Subtract the mean of elements of w , which corresponds to the projection of w onto W LC . 3.

Original: w w= k w k : Normalize w to a unit vector.
Revised: w w= k w k : Normalize w to a unit vector.

Original: (cid:24) (cid:24) U (0 ; 1) :
Revised: (cid:24) (cid:24) U (0 ; 1) :

Original: Generate a sample (cid:24) from a uniform distribution in (0 ; 1) .
Revised: Generate a sample (cid:24) from a uniform distribution in (0 ; 1) .

Original: w (cid:24) 1 m (cid:0) 2 w : Shrink w in proportion to the hypersurface area of the m (cid:0) 1 dimensional hypersphere of radius (cid:24) . 
Revised: w (cid:24) 1 m (cid:0) 2 w : Shrink w in proportion to the hypersurface area of the m (cid:0) 1 dimensional hypersphere of radius (cid:24) . [Figure]

Original: B A DDITIONAL GRAPHS
Revised: B A DDITIONAL GRAPHS

Original: The convergence and computation time for training MLPs with SVHN and CIFAR-100 datasets are shown in Figure 14 and Figure 15, respectively.
Revised: The convergence and computation time for training MLPs with SVHN and CIFAR-100 datasets are shown in Figure 14 and Figure 15, respectively.

Original: In each ﬁgure, (a)-(c) shows results for MLP (100 ; 128) , (d)-(f) for MLP (50 ; 256) , and (g)-(i) for MLP (5 ; 512) .
Revised: In each ﬁgure, (a)-(c) shows results for MLP (100 ; 128) , (d)-(f) for MLP (50 ; 256) , and (g)-(i) for MLP (5 ; 512) .

Original: We see that both LCW and BN enable training deep models, such as MLP (100 ; 128) and MLP (50 ; 256) .
Revised: We see that both LCW and BN enable training deep models, such as MLP (100 ; 128) and MLP (50 ; 256) .

Original: LCW achieves faster convergence with respect to the training accuracy in most cases.
Revised: LCW achieves faster convergence with respect to the training accuracy in most cases.

Original: [Figure]
Revised: [Figure]

Original: [Figure]