P ROPER S TRAIGHT T HROUGH E STIMATOR : B REAKING SYMMETRY PROMOTES CONVERGENCE TO TRUE MINIMUM
Anonymous authors
Paper under double-blind review
A BSTRACT
In the quantized network, its gradient shows vanishing except for nondifferentiable points.
The network thus cannot be learned by the standard backpropagation, so that an alternative approach called Straight Through Estimator (STE), which replaces the part of the gradient with a simple differentiable function, is used.
While STE is known to work well for learning the quantized network empirically, it has not been established theoretically.
A recent study by Yin et al.
(2019) has provided theoretical support for STE.
However, its justiﬁcation is still limited to the model in the one-hidden layer network with the binary activation where Gaussian generates the input data, and the true labels are output from the teacher network with the same binary network architecture.
In this paper, we discuss the effectiveness of STEs in more general situations without assuming the shape of the input distribution and the labels.
By considering the scale symmetry of the network and speciﬁc properties of the STEs, we ﬁnd that STE with clipped Relu is superior to STEs with identity function and vanilla Relu.
The clipped Relu STE, which breaks the scale symmetry, may pick up one of the local minima degenerated in scales, while the identity STE and vanilla Relu STE, which keep the scale symmetry, may not pick it up.
To conﬁrm this observation, we further present an analysis of a simple misspeciﬁed model as an example.
We ﬁnd that all the stationary points are identical with the vanishing points of the cRelu STE gradient, while some of them are not identical with the vanishing points of the identity and Relu STE.
Finally we have numerically conﬁrmed the observation for the mixture Gaussian model with various teacher network.
I NTRODUCTION
Quantization of the weights and the activations is a promising technique to save the memory and accelerate the inference speed in deep neural networks (DNNs) which have been getting wider and deeper in recent years.
There are two main approaches of the quantization for DNNs (Krishnamoorthi (2018)):
Post-training quantization (PTQ) and quantization-aware training (QAT).
In the PTQ, the pre-trained networks are simply quantized without re-training the model.
This approach allows us to achieve the nearly ﬂoating point accuracy at 8-bits, while below 8-bits, this results in signiﬁcant accuracy degradation.
Although there have been recent attempts to alleviate the accuracy degradation in PTQ (Banner et al.
; Choukroun et al.
; Zhao et al.
; Kryzhanovskiy et al.
(2021)), the QAT, where the quantized weights and activations are trained (e.g., Zhou et al.
Hubara et al.
; Rastegari et al.
(2016)), usually leads to better accuracy.
The difﬁculty in QAT is that the weights and the activations are discretized, and intrinsically nondifferentiable.
If we take the derivatives forcibly, they either vanish or diverge.
To avoid this problem, we replace them with the derivatives of some differentiable function in the backward pass only, called the Straight-Through Estimator (STE) (Hinton et al.
; Bengio et al.
Since the replacement leads to bias, it is not always possible to learn the network successfully. However this approach can be applied to the low bits below 8-bits with tolerable accuracy degradation (Zhou et al.
; Choi et al.
Esser et al.
; Bhalgat et al.
Originally, STE was introduced as the replacement with the derivative of the identity function by Hinton et al. Later, the term ”STE” has been extensively used as the replacement with various functions.
In the binary network, Bengio et al.
(2013) studied the replacement with the derivative of sigmoid function as well as the original STE, and Hubara et al.
(2017) used the derivative of the identity function clipped in the region of | x | ≤ 1 .
In low-bits network, Zhou et al.
(2016) used the quantized gradient after the replacement with the identity function, and Choi et al.
(2018), Esser et al.
(2019), and Bhalgat et al.
(2020) used the the derivative of the clipped Relu (cRelu) for leaning the step size.
1. M AIN CONTRIBUTION
In this paper, we refer to the proxy of the gradient as the STE gradient 1 .
Although STE has been much success in training quantized DNNs empirically, the theoretical justiﬁcation is very limited to the speciﬁc situations.
It also remains unclear what differentiable functions should be used for the
STE gradient.
Our aim is to shed light on a new factor to determine the functions.
Without assuming the shape of the input distribution and the loss function, we ﬁrst discuss the properties of the three STEs, identity STE, Relu STE, and cRelu STE in one-hidden-layer network with binary activation from the perspective of the intrinsic symmetry.
We ﬁnd that because the identity STE and Relu STE keep the scale symmetry, the STE gradients should be zero for any scale to converge to the true (local) minimum of the loss function, while in the case of the cRelu STE, which breaks the scale symmetry, even if its gradient is not zero at the minimum at some scale and it becomes zero at other scale, the cRelu STE converges to the minimum.
Therefore, the backpropagation using the cRelu STE is most likely to converge to the minimum of the loss function among three STEs.
This result reveals differences between Relu STE and cRelu STE, which could not be found in the models by Yin et al.
(2019) and Long et al.
Recently, Kunin et al.
(2020) discussed the symmetry embedded in non-quantized neural networks, and the effect of its breaking such as the discretization, the weight decay, and the stochasticity during training.
We discuss the new effect of breaking symmetry by the proxy of the gradient in training the quantized neural networks.
Next, to conﬁrm this observation, we have studied a similar model of the Gaussian input as discussed in Yin et al.
as an example.
We have employed a misspeciﬁed model suitable for most practical situations.
Unlike the previous study, the true labels are presumed to be generated by the non-quantized network architecture.
Consequently, we ﬁnd that ignoring the scale degeneracy for the weight in front of the activation, all the stationary points are identical with the vanishing points in the cRelu STE gradient, while some of them are not identical with the vanishing points in the identity and the Relu STE gradients.
In particular, at the global minimum point, both identity and
Relu STE gradients do not vanish.
Finally, to conﬁrm it in more general case, we have numerically studied the model of the mixture Gaussian input.
1. R ELATED W ORKS
Recently, there have been a few theoretical studies on the justiﬁcation of the STEs.
Shekhovtsov &
Yanush (2020) derived STE by the linearization of their proposed estimator in a stochastic binary deep network where the noises are injected before the binary activation to be a smooth network.
Cheng et al.
(2019) argued that STE can be interpreted as a projected Wasserstein gradient ﬂow under certain conditions.
Yin et al.
(2019) that inspired our study examined which of the three
STE’s, identity STE, Relu STE, and cRelu STE, converges to the true minimum in one-hiddenlayer network with binarized activation and Gaussian input data.
They clariﬁed that if the labels are obtained from a teacher network of the same architecture (Du et al.
, all the three STEs show the non-negative correlation with the population gradient, but only the identity STE gradient does not become zero in a local minimum.
This indicates that the back-propagation using either
Relu STE or cRelu STE may convergence to the local minimum, while it is impossible to show the convergence to the local minimum using the identity STE.
Furthermore, they showed that all the three STEs gradients vanish at the global minimum, indicating that they may achieve the global minimum independent of the choice of the STE, if we start with an appropriate initial value.
Very recently, Long et al.
(2021) have discussed the justiﬁcation of a class of STEs with certain monotonicity, which generalizes the Relu STE, in one-hidden-layer network for the hinge loss, claiming that the STE gradients vanish at the global minimum.
1. N OTATIONS
|·| denotes the Euclidean norm of a vector.
A capital letter in bold denotes the matrix, e.g., Z , and its component is given by the letter with lower indices, e.g., Z ij .
A small letter in bold denotes the vector, e.g., g , and its component is given by the letter with lower indices, e.g., g j .
I m ×
m is the m × m identity matrix, and 1 m is the m -dimensional vector of all ones.
G ENERAL DISCUSSION ON STE S IN ONE - HIDDEN LAYER CNN WITH BINARY ACTIVATION 2. PRELIMINARIES
The one-hidden layer convolutional neural network with a binary activation function is realized as
[Equation]
where σ ( · ) is the step function,
[Equation] Z ∈ R m × n is constructed by deividing the input data, e.g. an image, into m patches with size n , and w ∈ R n and v ∈ R m are the trainable weights, i.e. kernels, in the ﬁrst and second layers, respectively (Du et al.
We assume that Z is generated by a continuous distribution P ( Z ) .
Due to the property of the step function, the network is invariant under the scale transformation for w , w → k w ( k > 0) .
Using the loss for each sample, (cid:96) ( y ( v , w , Z ); y ∗ ) with y ∗ being the labels generated by the true distribution q ( y ∗ | Z ) , we can express the population loss function as
[Equation]
2. B ACK - PROPAGATION AND STE GRADIENTS
The gradients of the loss for each sample w.r.t v and w can be formally expressed as
[Equation]
where x i = σ ( (cid:80)
j Z ij w j ) and we used the property that the derivative of the step function is Dirac’s delta function, dσ ( z ) /dz
= δ ( z ) .
The gradient w.r.t v can be utilized for learning the network, while it is impossible to use the gradient w.r.t w due to the presence of the delta function.
Remarkably, both population gradients w.r.t v and w are smooth functions due to its average over the continuous distribution,
[Equation]
where we used the orthogonal transformation, ˜ Z ik (cid:48) = (cid:80) j (cid:48) O ik (cid:48) j (cid:48)
Z ij (cid:48) , Z ij (cid:48) =
(cid:80) k (cid:48) O ik (cid:48) j (cid:48) ˜
Z ik (cid:48) , with O i 1 j (cid:48) =
w j (cid:48) / | w | and (cid:80) j (cid:48)
O ik (cid:48)
j (cid:48)
O il (cid:48)
j (cid:48) =
δ k (cid:48) ,l (cid:48) .
We can easily see that the gradient w.r.t.
w does not have the component in w -direction, (cid:80) j w j E Z (cid:104)
∂(cid:96)∂w j (cid:105)
= 0 .
Its vanishing originally follows from the scale symmetry of the network: Because the loss has the scale symmetry, its gradient in the w -direction is obviously zero.
Instead of the population gradient, we use the STE in practice, which replaces the delta function with the derivative of some differentiable function µ STE , to train the kernel w ,
[Equation]
We consider three types of µ STE ( x ) : (i) the identity type µ STE ( x ) = x , (ii) vanilla ReLU type µ STE ( x )
= xσ ( x ) and (iii) the cReLU type µ STE ( x )
= xσ ( x ) σ ( r − x ) with the upper clipping value r .
The surrogate back-propagation using the STE is described in Algorithm 1 (Yin et al. layer CNN.
Input : initialization v 0 ∈ R m , w 0 ∈ R n , learning rate η . for t = 0 , 1 , , . . .
do
[Equation]
end for
The average of the STE gradients over the input data is also expressed as
[Equation]
Note that in general, the STE gradients have the non-zero component in the w -direction due to their partial collapse of the form of the gradient, (cid:80) j w j E Z (cid:2) g STE j
[Equation]
It is obvious from Eq.(9) that the only difference between the STE gradients with the identity function, Relu, and cRelu is the region of the integration over ˜ Z
i 1 ,
[Equation]
where A = id , Relu , cRelu , and the integration reigion R A are given as R id = ( −∞ , + ∞ ) , R Relu = [0 , + ∞ ) , and R cRelu = [0 , r | w | ] .
Following the scale invariance of F ij ( v , w , ˜
Z i 1 ) for w , F i (cid:48) j ( v , k w , ˜ Z i (cid:48) 1 ) = F i
[Equation]
j ( v , w , ˜ Z i (cid:48) 1 ) ( k > 0) , the STE gradients have the following interesting features:
The identity and Relu STE gradients have scale invariance for w ,
[Equation]
The cRelu STE gradient does not have scale invariance for w due to the clipping effect,
[Equation]
where we have explicitly shown the dependence of the upper clipping value r .
Instead, the scale transformation of w for the cRelu STE gradient can be compensated by that of r :
[Equation]
If we take kr with ﬁxed r large enough to cover most of the distribution P ( Z ) , the left-hand side is identical with the Relu STE gradient.
Here we deﬁne the spread of the distribution as ρ .
When we redeﬁne w /k as w on the right hand, the cRelu STE gradient for | w | < r/ρ is approximately identical with the Relu STE gradient,
[Equation]
This result is intuitively obvious: when w is small enough to keep most of the pre-activation values in the clipped range, the cRelu gradient and the Relu gradient are approximately identical.
The important issue is that there is no guarantee that the weights obtained by the STE backpropagation are the ones at the (local) minimum of the loss function.
In other words, at the stationary points, ( v , w ) =
( v s , w s ) , deﬁned as the vanishing points of the population gradient,
[Equation]
the STE gradient is not zero in general.
Here we consider the local minimum point that the STE gradient does not vanish,
[Equation]
Note that ( v , w ) = ( v s , k w s ) for any k > 0 is also the local minimum point due to the scale invariance of the loss. In the case of the Relu/identity STEs, their gradient has the same ﬁnite value for any scales due to the scale invariance shown in Eq. [Equation]
[Equation] In the case of the cRelu STE, however, the value of its gradient changes as the scale changes due to the breaking of the scale symmetry shown in Eq.(12). If it crosses zero as a function of scale,
[Equation]
some
[Equation]
the back propagation using cRelu STE can converge at the local minimum as was done by using the population gradient.
Therefore, we conjecture that due to breaking scale symmetry, the backpropagation using cRelu STE is the most likely to achieve the (local) minimum of the loss function in the three STEs. This also implies that the cRelu STE is less biased than the other.
We conﬁrm the conjecture analytically for the Gaussian input with Relu-type teacher network in Sec.3, and numerically for the mixture Gaussian input with various teacher network in Sec. 4.
Since the above discussion is focused only on symmetry, the conjecture can be generalized to networks with more symmetries in a straightforward way.
In that case, if we employ an STE that breaks all of those symmetries instead of cRelu STE, it is more likely to achieve the minimum.
The similar mechanism can be also found in physical phenomena.
For instance, this is the case of the ferromagnet, which has rotational symmetry in energy (or Hamiltonian).
In the system, if we impose the magnetic ﬁeld, which breaks the rotational symmetry, the spin states, corresponding to minimum points, become aligned with the same direction.
O NE - HIDDEN LAYER CNN WITH BINARY ACTIVATION : G AUSSIAN INPUT AND LABELS GENERATED BY NON - QUANTIZED R ELU NETWORK
We consider the simple model similar to Yin et al.
(2019) given by the Gaussian input with the variance ˆ σ 2 , Z ∼ P ( Z ) =
[Equation]
We use the squared loss function,
[Equation]
( y ( v , w , Z ); y ∗ ( v ∗ , w ∗ , Z ))
= 12 ( y − y ∗ ) 2 .
Unlike in Yin et al.
, the true labels are assumed to be generated by the non-quantized Relu network,
[Equation]
with the Relu activation deﬁned as f Relu ( x ) = xσ ( x ) .
Note that the models are misspeciﬁed, i.e., the population loss cannnot be zero even at the global minimum.
In fact, it is inferred from the accuracy degradation in most practical situations (Hubara et al.
that the binary networks are considered as misspeciﬁed models.
On the other hand, if the true labels are generated by the same binarized architecture as was used in Yin et al.
(2019), the network becomes a speciﬁed model.
As shown below, this misspeciﬁed model leads to a more striking difference between three STEs than the speciﬁed model.
Particularly, the behaviors of three STE gradients at the global minimum are completely different from each other.
3. POPULATION LOSS , ITS GRADIENT AND STATIONARY POINTS
The population loss L ( v , w ; v ∗ , w ∗ ) is derived in Appendix A.1 as
[Equation]
( v , w ; v [Equation] , w [Equation]
Z [
[Equation]
Z
[Equation]
(20) where ϕ is the angle between w ∗ and w .
Note that the loss is scale invariant for w , and thus its derivative satisﬁes e w · ∂L∂ w = 0 .
The stationary points are given by
[Equation]
where we used (cid:0) I m
[Equation]
1 m
[Equation]
(cid:1)
[Equation]
There are three stationary points 2 as classiﬁed in Appendix A.2: 1.
middle saddle point given by v = ¯
v ≡ 2ˆ
[Equation]
w
[Equation]
| [Equation]
Tm
[Equation] v
[Equation] [Equation]
(cid:21) , which satisﬁes ¯ v T v ∗ = 0 .
This exists if and only if ( m + 1) ( v
[Equation]
[Equation]
2 (cid:0) v
[Equation]
T 1 m (cid:1) 2 . local minimum if ( m + 1) ( v ∗ ) 2 ≥ 2 (cid:0) v
[Equation]
T 1 m (cid:1) 2 , otherwise saddle point given by v =
[Equation]
global minimum given by v = v 0 ≡ 2ˆ σ | w ∗ | √ 2 π v ∗ , ϕ = 0 .
At the global minimum, the population loss does not become zero, L ( v = v 0 , w = w ∗ ) = ˆ σ 2 | w ∗ |
[Equation]
π − 2) 4 π v ∗ T v .
3. STE GRADIENT
The STE gradient is given by
[Equation]
    [Equation] The converging points by the back-propagation in Algorithm 1 are characterized by vanishing the gradients,
[Equation]
We discuss the explicit form of the three STE gradients and the associated vanishing points.
The detailed derivation of the STE gradients is shown in Appendix A.3-A.5.
3.2. IDENTITY STE
The identity STE gradient is given by
[Equation]
Combined with Eq.(21), we ﬁnd that the equation in Eq.(24) has no solution.
Therefore, if we use the back-propagation by identity STE, it can not converge to any points.
3.2. R ELU STE
The Relu STE gradient is expressed as
[Equation]
Combined with Eq.(21), the gradients vanish at the following points:
If v ∗ satisﬁes ( m + 1) ( v ∗ ) 2 ≥ 2 (cid:0) v ∗ T 1 m (cid:1)
[Equation]
We ﬁnd the solutions are identical with two stationary points shown in item 1 and 2 in Sec.3.1, leading to the the saddle point or local minimum, while the global minimum point at ϕ = 0 , v = vcannot be obtained.
Therefore, by using the back-propagation with Relu STE, it always converges to ϕ
=
π if ( m + 1) ( v ∗ ) 2 ≥ 2 (cid:0) v ∗ T 1 m (cid:1)
[Equation]
and it does not converge to any points, otherwise.
3.2. CLIPPED R ELU STE
The cRelu STE is given by
[Equation]
27) where C ( | w | , ϕ ) and S ( | w | , ϕ ) are given in Eq.(106). At ϕ = 0 , π they are simpliﬁed as
[Equation]
Remarkably, the cRelu STE gradient is proportional to the Relu STE gradient in the case of v T v ∗ = 0 or ϕ
= π :
[Equation]
As shown in Sec.3.2.2, all the vanishing points in the Relu STE can be found at v T v ∗ = 0 or ϕ
= π , leading to the saddle point or the local minimum, so that they are also found in the vanishing points of cRelu STE.
On the other hand, the behavior of cRelu STE gradient at ϕ = 0 is not related to that of Relu STE gradient.
The cRelu STE gradient at ϕ = 0 is written as
[Equation]
where
[Equation]
This becomes zero if
[Equation]
To be consistent with Eq.(21) at ϕ = 0 , only λ = 2 is allowed.
The solution provides the global minimum shown in item 3 in Sec.3.1.
In fact, λ monotonically increases from 0 to π as [Equation] and thus the solution at λ = 2 , corresponding to the global minimum point, can be obtained for any clipping value r and the variance ˆ σ 2 by changing the scale | w | .
Consequently, we get all the stationary points are found in cRelu STE:
If v ∗ satisﬁes [Equation] + [Equation]
[Equation]
ϕ
= 0 , | w [Equation] = [Equation]
≡
[Equation] ˆ σ c 0 , v = v 0 ≡ 2ˆ σ | w ∗ | [Equation] 2 π v ∗ , where w 0 (or c 0 ) is given by λ ( w 0 )
= 2 .
Note that while the global minimum point is degenerated in scales for w , i.e. if ( w , v ) =
( w 0 , v 0 ) gives the global minimum, ( w , v ) =
( k w 0 , v 0 ) for any k > 0 also gives the global minimum, the cRelu STE picks up the point at the scale determined by λ = 2 .  E XPERIMENTS
To conﬁrm our conjecture in more general case, we have numerically studied the Gaussian mixture input with various mean values.
As teacher networks, we have tested tanh-type and sin-type as well as Relu-type.
For all the examined setups, cRelu STE shows the similar behavior to the population gradient, while id/Relu STEs and cRelu STE shows qualitatively different behaviors as described below. To calculate the population loss and STE gradients, we have generated the mixture Gaussian samples and have taken their average. The population gradient has been obtained by calculating the ﬁnite difference of the population loss. We have demonstrated the back propagation with learning rate η = 0 .
01 given in Algorithm 1.
Shown in Fig.
1 are the results of ten mixture Gaussian input with random mean values for each components of Z .
We employed m = 20 , n = 25 , and the tanh-type teacher network.
We ﬁnd that the population gradient and cRelu STE show similar results, while id/Relu STE are completely different.
This indicates cRelu STE is the less biased estimator than the other estimators.
In the case of id/Relu STEs, at early steps up to around 500 step, | w | decreases around the magnitude of the update quantity, so that it begins to oscillate.
Then it escapes the oscillation, and the loss function shows the convergence to a point different from the local minimum achieved by population gradient, while | w | becomes larger and larger due to their scale invariance.
Interestingly, the values of the loss function are small compared to the one obtained by population gradient, which implies that id/Relu STEs avoid being trapped in the local solution due to their large bias.
However, note that even at that point, the magnitude of w continues to grow and eventually become numerically unstable.
We have found that breaking symmetry embedded in the network by STEs enhances the possibility of convergence to the true (local) minimum of the loss function.
We have demonstrated that if an STE breaks the scale symmetry embedded in the one-hidden-layer network with a binary activation, it is more likely to achieve the local minimum than the one which keeps the symmetry.
The discussion can be generalized to the network with more symmetries in a straightforward way.
The more symmetries embedded in the network an STE breaks, the more likely it is to converge.
To conﬁrm the mechanism, we have studied three STEs, identity STE, Relu STE, and cRelu STE, in a simple misspeciﬁed model with Gaussian input.
We have found that the back-propagation by the cRelu STE, which breaks the scale symmetry, can converge to the global minimum, while identy/Relu STE cannot.
A DERIVATION OF S EC .
In the Appendix, we provide the results of the Gaussian with variance 1, N (0 , 1) .
For some function f ( Z ) , the expectation value of the Gaussian with the generic variance ˆ σ, N (0 , ˆ σ 2 ) , shown in Sec. is written as
[Equation]
where we have changed the integral variables from Z to Z (cid:48)
=
Z /σ .
Therefore we get the results with the variance ˆ σ by only changing from f ( Z ) to f (ˆ σ Z ) .
A. DERIVATION OF LOSS FUNCTION
The expectaion of the loss is divided into the following three terms:
[Equation]
The calculation of the ﬁrst term and the third term are given in “Proof of Lemma 1” in Yin et al.
(2019) and “Proof of Section 3” in Ref.
Du et al.
(2018), respectively:
[Equation]
To discuss the second term
[Equation]
we evaluate the following quantity:
[Equation]
where we used the orthogonal transformation in the second line:
[Equation]
with O i 1 m (cid:48)
=
w m (cid:48) / | w | and O j 1 m
[Equation]
[Equation]
(ii)
[Equation]
case:
[Equation]
Without loss of generality [Equation] we can choose
[Equation]
with w ∗ > [Equation] so that this can be written as
[Equation] [Equation]
where ϕ is the angle between w ∗ and w .
Eqs.(36)
and (40) are combined as
[Equation]
so that the second term in Eq.
(33) is expressed as
[Equation] [Equation]
Consequently, the loss function is summarized as
[Equation]
The population gradient w.r.t v and w is thus given by
[Equation]
where without loss of generality
[Equation]
we can take e w = (cos ϕ, sin ϕ, 0 n − 2 ) , e ϕ =
( − sin ϕ, cos ϕ, 0 n − 2 ) .
Note that e w · ∂L∂ w = 0 is satisﬁed due to the scale symmetry for the loss.
A. CLASSIFICATION OF STATIONARY POINTS
Using Eq.(21)
[Equation]
the loss at the three stationary points is rewritten as
[Equation] [Equation]
Obviously, this shows that ϕ
= 0 is the global minimum point,
[Equation]
[Equation]
At ϕ = π , this becomes
[Equation]
At ϕ = ¯ ϕ = arccos
[Equation]
, this becomes
[Equation]
Using these results, we obtain
[Equation]
Note that because ( m + 1) ( v ∗ ) 2 ≥ 2 (cid:0) v ∗ T 1 m (cid:1) 2 is satisﬁed to exist the stationary point at ϕ = ¯ ϕ , the loss at ϕ = π is smaller than the one at at ϕ = ¯ ϕ ,
[Equation]
51) is satisﬁed.
When ( m + 1) ( v ∗ ) 2 = 2 (cid:0) v ∗ T 1 m (cid:1) 2 , L | ϕ =¯ ϕ
= L | ϕ
= π , because the middle stationary point at ϕ =
¯ ϕ is merged into the point at ϕ = π .
This implies that ϕ
= 0 is the global minimum, ϕ
= π is the local minimum, and ϕ = ¯
ϕ is the saddle point or the local maximum if ( m + 1) ( v ∗ ) 2 ≥ 2 (cid:0) v ∗ T 1 m (cid:1) 2 .
To fully clarify whether the stationary points are local minimum, maximum or saddle point, we have calculated the hessian matrix,
[Equation]
[Equation]
For z = ( x , y ) T ( x ∈ R m , y ∈ R ) ,
[Equation]
where we used ¯ v T v ∗ = 0 .
Therefore, depending on x and y , the Hessian can be either positive or negative, which means the stationary point is a saddle point.
[Equation]
Note that v
Tπ v ∗ = 2 |
w ∗ | √ 2 π v ∗ T (cid:16) − I m
× m + 2 m +1 1 m 1 Tm (cid:17) v ∗ .
If ( m + 1) ( v ∗ ) 2 ≥ 2 (cid:0) v ∗ T 1 m (cid:1) 2 , then v Tπ v ∗ ≤ 0 , so that the matrix becomes a positive deﬁnite matrix: For any x ∈ R m , y ∈ R , this satisﬁes
[Equation]
(55) which means the stationary point is the local minimum.
On the other hand, if ( m + 1) ( v ∗ ) 2 ≤ 2 (cid:0) v ∗ T 1 m (cid:1) 2 , the matrix becomes either positive or negative, depending on x and y , so that this stationary point becomes a saddle point.
[Equation]
The Hessian becomes a positive deﬁnite matrix: For any x ∈ R m , y ∈ R , this satisﬁes
[Equation]
A. DERIVATION OF IDENTITY STE GRADIENT
By substituting µ (cid:48) ( x ) = 1 into Eq.(23), we obtain
[Equation]
We calculate each term as follows.
A.3. CALCULATION OF THE FIRST TERM
We calculate the ﬁrst term as was done in Lemma 9 (and also Lemma 11) in Yin et al.
To discuss the ﬁrst term, we evaluate
[Equation]
[ σ ( Zw )] k ] .
Using F id ijk , the ﬁrst term can be expressed as ﬁrst term =
(cid:88)
[Equation] [Equation]
(i) j (cid:54) =
k case:
[Equation]
because of the odd symmetry for i -th and j -th components in the integrand. (ii)
[Equation]
[Equation]
62) where we used the orthogonal transformation in the second line:
[Equation]
[Equation] with O i 1 m (cid:48)
=
w m (cid:48) / | w | .
The ﬁrst term is reduced to ﬁrst term
[Equation]
[Equation]
A.3. CALCULATION OF THE SECOND TERM
To discuss the second term, we evaluate
[Equation]
Using G id ijk , the second term can be expressed as
[Equation]
[Equation]
j (cid:54)
[Equation]
case
[Equation]
(67) because of the odd symmetry.
[Table]
[Equation]
) where we have used the orthogonal transformation in the second line:
[Equation] [Equation]
with O i 1 m (cid:48) =
w ∗ m (cid:48) / | w ∗ | , and also have used (cid:90) [Equation]
[Equation]
The second term is written as
[Equation]
A. DERIVATION OF R ELU STE GRADIENT
By substituting µ (cid:48) ( x ) = σ ( x ) into Eq.(23)
[Equation]
we obtain
[Equation]
We calculate each term as follows.
A.4. CALCULATION OF THE FIRST TERM
We calculate the ﬁrst term as was done in Lemma9 (and also Lemma11) in Yin et al.
We evaluate
[Equation]
Using F relu ijk , the ﬁrst term can be expressed as ﬁrst term =
(cid:88)
[Equation]
[Equation]
where we used the orthogonal transformation in the second line:
[Equation]
with O i 1 m (cid:48)
=
w m (cid:48) / | w | .
[Equation]
= k case:
We get the same expression given in Eq.(62),
[Equation]
The ﬁrst term is written as ﬁrst term
[Equation]
A.4.
CALCULATION OF SECOND TERM
To discuss the second term [Equation] we evaluate
[Equation]
Using G relu ijk [Equation] the second term can be expressed as second term
[Equation]
[Equation]
(i)
[Equation]
case:
[Equation] [Equation]
[Equation]
with O i 1 m (cid:48)
=
w m (cid:48) / | w | and O j 1 m (cid:48)
=
w ∗ m (cid:48) / | w ∗ | .
(ii) j = k case:
[Equation]
relu ijj
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
=
[Equation]
(cid:90)
[Equation]
[Table]
[Equation]
where we have chosen
[Equation]
with w ∗ > 0 and ϕ is deﬁned as the angle between w ∗ and w .
We also have used the formula,
[Equation]
Note that
[Equation]
[Equation]
This reduces to
[Equation]
In summary
[Equation]
the second term is written as
[Equation]
A. DERIVATION OF CLIPPED R ELU STE GRADIENT
By substituting µ (cid:48) ( x ) = σ ( x ) σ ( r − x ) into Eq.(23)
[Equation]
we obtain
[Equation]
We calculate each term as follows.
A.5. CALCULATION OF THE FIRST TERM
To discuss the ﬁrst term
[Equation]
we evaluate
[Equation]
Using F crelu ijk
[Equation]
the ﬁrst term can be expressed as ﬁrst
[Equation] [Equation]
[Equation]
where we used the orthogonal transformation in the second line:
[Equation]
with O i 1 m (cid:48)
=
w m (cid:48) / | w | .
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation]
In summary, the ﬁrst term is written as ﬁrst term =
[Equation]
A.5. CALCULATION OF THE SECOND TERM
To discuss the second term
[Equation]
we evaluate
[Equation]
Using G crelu ijk
[Equation]
the second term can be expressed as
[Equation] [Equation] [Table]
w m
[Table]
| w [Table] m (cid:48)
=
w
[Table] m (cid:48) [Table] | w
[Table] | [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where we have chosen
[Equation]
with w ∗ > 0 and ϕ is deﬁned as the angle between w ∗ and w .
We also have used the formula,
[Equation]
[Equation]
Note that
[Equation]
[Equation]
Using C ( w, ϕ ) , S ( w, ϕ ) deﬁned as
[Equation]
If ϕ (cid:54) = 0 , π , G crelu ijj is formally written as
[Equation]
If ϕ = 0 or π , G crelu ijj is formally written as
[Equation]
) where we used S ( w, 0)
= S ( w, π ) =
C ( w, π ) = 0 and
[Equation]
with the following formula
[Equation]
In summary
[Equation]
the second term is written as
[Equation]