O N THE C ONVERGENCE AND R OBUSTNESS OF B ATCH N ORMALIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite its empirical success, the theoretical underpinnings of the stability, convergence and acceleration properties of batch normalization (BN) remain elusive.
In this paper, we attack this problem from a modeling approach, where we perform a thorough theoretical analysis on BN applied to a simpliﬁed model: ordinary least squares (OLS).
We discover that gradient descent on OLS with BN has interesting properties, including a scaling law, convergence for arbitrary learning rates for the weights, acceleration effects, as well as insensitivity to the choice of learning rates.
We then demonstrate numerically that these ﬁndings are not speciﬁc to the OLS problem and hold qualitatively for more complex supervised learning problems.
This points to a new direction towards uncovering the mathematical principles that underlies batch normalization.
I NTRODUCTION
Batch normalization (Ioffe & Szegedy, 2015) (BN) is one of the most important techniques for training deep neural networks and has proven extremely effective in avoiding gradient blowups during back-propagation and speeding up convergence.
In its original introduction (Ioffe & Szegedy, 2015), the desirable effects of BN are attributed to the so-called “reduction of covariate shift”.
However, it is unclear what this statement means in precise mathematical terms.
To date, there lacks a comprehensive theoretical analysis of the effect of batch normalization.
In this paper, we study the convergence and stability of gradient descent with batch normalization (BNGD) via a modeling approach.
More concretely, we consider a simpliﬁed supervised learning problem: ordinary least squares regression, and analyze precisely the effect of BNGD when applied to this problem.
Much akin to the mathematical modeling of physical processes, the least-squares problem serves as an idealized “model” of the effect of BN for general supervised learning tasks.
A key reason for this choice is that the dynamics of GD without BN (hereafter called GD for simplicity) in least-squares regression is completely understood, thus allowing us to isolate and contrast the additional effects of batch normalization.
The modeling approach proceeds in the following steps.
First, we derive precise mathematical results on the convergence and stability of BNGD applied to the least-squares problem.
In particular, we show that BNGD converges for any constant learning rate ε ∈ (0 , 1] , regardless of the conditioning of the regression problem.
This is in stark contrast with GD, where the condition number of the problem adversely affect stability and convergence.
Many insights can be distilled from the analysis of the OLS model.
For instance, we may attribute the stability of BNGD to an interesting scaling law governing ε and the initial condition; This scaling law is not present in GD.
The preceding analysis also implies that if we are allowed to use different learning rates for the BN rescaling variables ( ε a ) and the remaining trainable variables ( ε ), we may conclude that BNGD on our model converges for any ε > 0 as long as ε a ∈ (0 , 1] .
Furthermore, we discover an acceleration effect of BNGD and moreover, there exist regions of ε such that the performance of BNGD is insensitive to changes in ε , which help to explain the robustness of BNGD to the choice of learning rates.
We reiterate that contrary to many previous works, all the preceding statements are precise mathematical results that we derive for our simpliﬁed model.
The last step in our modeling approach is also the most important: we need to demonstrate that these insights are not speciﬁc features of our idealized model.
Indeed, they should be true characteristics, at least in an approximate sense, of BNGD for general supervised learning problems.
We do this by numerically investigating the convergence, stability and scaling behaviors of BNGD on various datasets and model architectures.
We ﬁnd that the key insights derived from our idealized analysis indeed correspond to practical scenarios.
1. R ELATED WORK
Batch normalization was originally introduced in (Ioffe & Szegedy, 2015) and subsequently studied in further detail in (Ioffe, 2017).
Since its introduction, it has become an important practical tool to improve stability and efﬁciency of training deep neural networks (He et al., 2016; Bottou et al., 2018).
Initial heuristic arguments attribute the desirable features of BN to concepts such as “covariate shift”, which lacks a concrete mathematical interpretation and alternative explanations have been given (Santurkar et al., 2018).
Recent theoretical studies of BN includes (Ma & Klabjan, 2017), where the authors proposed a variant of BN, the diminishing batch normalization (DBN) algorithm and analyzed the convergence of the DBN algorithm, showing that it converges to a stationary point of the loss function.
More recently, (Bjorck et al., 2018) demonstrated that the higher learning rates of batch normalization induce a regularizing effect.
Another related work is (Kohler et al., 2018), where the authors also considered the convergence properties of BNGD on linear networks (similar to the least-squares problem), as well as other special problems, such as learning halfspaces and extensions.
In the OLS case, the authors showed that for a particularly adaptive choice of dynamic learning rate schedule, which can be seen as a ﬁxed effective step size in our terminology (see equation (11) and the discussion that immediately follows), BNGD converges linearly if λ max is known.
Moreover, the analysis also requires setting the rescaling parameter a every step to satisfy a stationarity condition, instead of simply performing gradient descent on a , as is done in the original
BNGD.
The present research differs from these previous analysis in an important way - we study the BNGD algorithm itself, and not a special variant.
More speciﬁcally, we consider constant learning rates (without knowledge of properties of the OLS loss function) and we perform gradient descent on rescaling parameters.
We prove that the convergence occurs for even in this case (and in fact, for arbitrarily large learning rates for ε , as long as 0 < ε a ≤ 1 ).
This poses more challenges in the analysis and contrasts our work with previous analysis on modiﬁed versions of BNGD.
This is an important distinction; While a decaying or dynamic learning rate is sometimes used in practice, in the case of BN it is critical to analyze the non-asymptotic, constant learning rate case, precisely because one of the key practical advantages of BN is that a bigger learning rate can be used than that in GD.
Hence, it is desirable, as in the results presented in this work, to perform our analysis in this regime.
Finally, through the lens of the least-squares example, BN can be viewed as a type of overparameterization, where additional parameters, which do not increase model expressivity, are introduced to improve algorithm convergence and stability.
In this sense, this is related in effect to the recent analysis of the implicit acceleration effects of over-parameterization on gradient descent (Arora et al., 2018).
1. O RGANIZATION
Our paper is organized as follows.
In Section 2, we outline the ordinary least squares (OLS) problem and present GD and BNGD as alternative means to solve this problem.
In Section 3, we demonstrate and analyze the convergence of the BNGD for the OLS model, and in particular contrast the results with the behavior of GD, which is completely known for this model.
We also discuss the important insights to BNGD that these results provide us with.
We then validate these ﬁndings on more general supervised learning problems in Section 4.
Finally, we conclude in Section 5.
B ACKGROUND
Consider the simple linear regression model where x ∈ R d is a random input column vector and y is the corresponding output variable.
Since batch normalization is applied for each feature separately, in order to gain key insights it is sufﬁcient to the case of y ∈ R .
A noisy linear relationship is assumed between the dependent variable y and the independent variables x , i.e. y =
x T w + noise where w ∈ R d is the parameters.
Denote the following moments:
[Equation]
To simplify the analysis, we assume the covariance matrix H of x is positive deﬁnite and the mean
E [ x ] of x is zero.
The eigenvalues of H are denoted as λ i ( H ) , i = 1 , 2 , ...
d, .
Particularly, the maximum and minimum eigenvalue of H is denoted by λ max and λ min respectively.
The condition number of H is deﬁned as κ := λ max λ min .
Note that the positive deﬁniteness of H allows us to deﬁne the vector norms (cid:107) .
(cid:107) H and (cid:107) .
(cid:107) H − 1 by (cid:107) x (cid:107) 2 H
=
x
T Hx and (cid:107) x (cid:107) 2 H − 1 = x T H − 1 x respectively.
2. O RDINARY LEAST SQUARES
The ordinary least squares (OLS) method for estimating the unknown parameters w leads to the following optimization problem
[Equation]
The gradient of J 0 with respect to w is ∇ w J 0 ( w ) =
Hw − g , and the unique minimizer is w = u :
= H − 1 g .
The gradient descent (GD) method (with step size or learning rate ε ) for solving the optimization problem (2) is given by the iterating sequence,
[Equation]
[Equation]
which converges if 0 < ε < 2 λ max =: ε max , and the convergence rate is determined by the spectral radius ρ ε :
= ρ ( I − εH )
= max i {| 1 − ελ i ( H ) |} with (cid:107)
[Equation]
It is well known (for example see Chapter 4 of (Saad, 2003)) that the optimal learning rate is ε opt =λ max + λ min , where the convergence estimate is related to the condition number κ ( H ) :
[Equation]
2. B ATCH NORMALIZATION
Batch normalization is a feature-wise normalization procedure typically applied to the output, which in this case is simply z = x T w .
The normalization transform is deﬁned as follows:
[Equation]
where σ := √ w T Hw .
After this rescaling, N BN ( z ) will be order 1, and hence in order to reintroduce the scale (Ioffe & Szegedy, 2015), we multiply N BN ( z ) with a rescaling parameter a
(Note that the shift parameter can be set zero since E [ w T x | w ] = 0 ).
Hence, we get the BN version of the
OLS problem (2):
[Equation] [Equation]
The objective function J ( a, w ) is no longer convex.
In fact, it has trivial critical points, { ( a ∗ , w ∗ ) | a ∗ = 0 , w ∗ T g = 0 } , which are saddle points of J ( a, w ) .
We are interested in the nontrivial critical points which satisfy the relations,
[Equation]
It is easy to check that the nontrivial critical points are global minimizers, and the Hessian matrix at each critical point is degenerate.
Nevertheless, the saddle points are strict (Details can be found in
Appendix), which typically simpliﬁes the analysis of gradient descent on non-convex objectives (Lee et al., 2016; Panageas & Piliouras, 2017).
Consider the gradient descent method to solve the problem (7), which we hereafter call batch normalization gradient descent (BNGD).
We set the learning rates for a and w to be ε a and ε respectively.
These may be different, for reasons which will become clear in the subsequent analysis.
We thus have the following discrete-time dynamical system
[Equation]
We now begin a concrete mathematical analysis of the above iteration sequence.
M ATHEMATICAL ANALYSIS OF BNGD ON OLS
In this section, we discuss several mathematical results one can derive concretely for BNGD on the OLS problem (7).
First, we establish a simple but useful scaling property, which an important ingredient in allowing us to prove a linear convergence result for arbitrary constant learning rates.
We also derive the asymptotic properties of the “effective” learning rate of BNGD (to be precisely deﬁned subsequently), which shows some interesting sensitivity behavior of BNGD on the chosen learning rates.
Detailed proofs of all results presented here can be found in the Appendix.
3. S CALING PROPERTY
In this section, we discuss a straightforward, but useful scaling property that the BNGD iterations possess.
Note that the dynamical properties of the BNGD iteration are governed by a set of numbers, or a conﬁguration { H, u, a 0 , w 0 , ε a , ε } .
Deﬁnition 3.1 (Equivalent conﬁguration) .
Two conﬁgurations, { H, u, a 0 , w 0 , ε a , ε } and { H (cid:48) , u (cid:48) , a (cid:48) 0 , w (cid:48) 0 , ε (cid:48)
a , ε (cid:48) } , are said to be equivalent if for iterates { w k } , { w (cid:48)
k } following these conﬁgurations respectively, there is an invertible linear transformation T and a nonzero constant t such that w (cid:48) k
= Tw k , a (cid:48) k
= ta k for all k .
The scaling property ensures that equivalent conﬁgurations must converge or diverge together, with the same rate up to a constant multiple.
Now, it is easy to check the system has the following scaling law.
Proposition 3.2 (Scaling property) .
Suppose µ (cid:54) = 0 , γ (cid:54) = 0 , r (cid:54) = 0 , Q T Q
=
I , then
(1) The conﬁgurations { µQ T HQ, γ √ µ Qu, γa 0 , γQw 0 , ε a , ε } and { H, u, a 0 , w 0 , ε a , ε } are equivalent.
The conﬁgurations { H, u, a 0 , w 0 , ε a , ε } and { H, u, a 0 , rw 0 , ε a , r 2 ε } are equivalent.
It is worth noting that the scaling property (2) in Proposition 3.2 originates from the batchnormalization procedure and is independent of the speciﬁc structure of the loss function.
Hence, it is valid for general problems where BN is used (Lemma A.9).
Despite being a simple result, the scaling property is important in determining the dynamics of BNGD, and is useful in our subsequent analysis of its convergence and stability properties (see the sketch of the proof of Theorem 3.3).
3. B ATCH NORMALIZATION CONVERGES FOR ARBITRARY STEP SIZE
We have the following convergence result.
Theorem 3.3 (Convergence for BNGD) .
The iteration sequence ( a k , w k ) in equation (9)-(10) converges to a stationary point for any initial value ( a 0 , w 0 ) and any ε > 0 , as long as ε a ∈ (0 , 1] .
Particularly, we have the following sufﬁcient conditions of converging to global minimizers.
(1) If a 0 w T 0 g > 0 (or a 0 = 0 , w T 0 g (cid:54) = 0 ), ε a ∈ (0 , 1] and ε is sufﬁciently small (the smallness is quantiﬁed by Lemma A.13), then ( a k , w k ) converges to a global minimizer.
(2) If ε a = 1 and ε > 0 , then ( a k , w k ) converges to global minimizers for almost all initial values ( a 0 , w 0 ) .
Sketch of Proof.
We ﬁrst prove that the algorithm converges for any ε a ∈ (0 , 1] and small enough ε , with any initial value ( a 0 , w 0 ) such that (cid:107) w 0 (cid:107) ≥ 1 (Lemma A.13).
Next, we observe that the sequence {(cid:107) w k (cid:107)} is monotone increasing, and thus either converges to a ﬁnite limit or diverges.
The scaling property is then used to exclude the divergent case if {(cid:107) w k (cid:107)} diverges, then at some k the norm (cid:107) w k (cid:107) should be large enough, and by the scaling property, it is equivalent to a case where (cid:107) w k (cid:107)
=1 and ε is small, which we have proved converges.
This shows that (cid:107) w k (cid:107) converges to a ﬁnite limit, from which the convergence of w k and the loss function value can be established, after some work.
The proof is fully presented in Theorem A.17 and preceding Lemmas.
In addition, using the ’strict saddle point’ arguments in (Lee et al., 2016; Panageas & Piliouras, 2017), we can prove the set of initial value for which ( a k , w k ) converges to saddle points has
Lebesgue measure 0, provided some conditions, such as when ε a = 1 , ε > 0 (Lemma A.20).
It is important to note that BNGD converges for all step size ε > 0 of w k , independent of the spectral properties of H .
This is a signiﬁcant advantage and is in stark contrast with GD, where the step size is limited by λ max ( H ) , and the condition number of H intimately controls the stability and convergence rate.
Although we only prove the almost sure convergence to global minimizer for the case of ε a = 1 , we have not encountered convergence to saddles in the OLS experiments even for ε a ∈ (0 , 2) with initial values ( a 0 , w 0 ) drawn from typical distributions.
3. C ONVERGENCE RATE , ACCELERATION AND ASYMPTOTIC SENSITIVITY
Now, let us consider the convergence rate of BNGD when it converges to a minimizer.
Compared with GD, the update coefﬁcient before Hw k in equation (10) changed from ε to a complicated term which we named as the effective step size or learning rate ˆ ε k
[Equation]
(11) and the recurrence relation in place of u − w k is
[Equation]
Consider the dynamics of the residual e k := u − ( w Tk g/σ 2 k ) w k , which equals 0 if and only if w k is a global minimizer.
Using the property of H -norm (see section A.1), we observe that the effective learning rate ˆ ε k determines the convergence rate of e k via
[Equation]
where ρ ( I − ˆ ε k H ) is spectral radius of the matrix I − ˆ ε k H .
The inequality (13) shows that the convergence of e k (and hence the loss function, see Lemma A.23) is linear provided ˆ ε k ∈ ( δ, 2 /λ max − δ ) for some positive number δ .
In fact, if we enforce ˆ ε k = 1 /λ max for each k , which is done in the analysis in Kohler et al.
(2018), then one immediately obtains the same linear convergence rate.
But this requires knowledge of λ max (problem-dependent) and a modiﬁcation the BNGD algorithm.
We instead focus our analysis on the original BNGD algorithm.
Next, let us discuss below an acceleration effect of BNGD over GD.
When ( a k , w k ) is close to a minimizer, we can approximate the iteration (9)-(10) by a linearized system.
The Hessian matrix for
BNGD at a minimizer ( a ∗ , w ∗ ) is diag (1 , H ∗ / (cid:107) w ∗ (cid:107) 2 ) , where the matrix H ∗ is
[Equation]
The matrix H ∗ is positive semi-deﬁnite ( H ∗ u = 0 ) and has better spectral properties than H , such as a lower pseudo-condition number κ ∗ =
λ ∗ max λ ∗ min ≤ κ , where λ ∗ max and λ ∗ min are the maximal and minimal nonzero eigenvalues of H ∗ respectively.
Particularly, κ ∗ < κ for almost all u (see section A.1 ).
This property leads to acceleration effects of BNGD: When (cid:107) e k (cid:107) H is small, the contraction coefﬁcient ρ in (13) can be improved to a lower coefﬁcient.
More precisely, we have the following result:
Proposition 3.4.
For any positive number δ ∈ (0 , 1) , if ( a k , w k ) is close to a minimizer, such that λ max ε | a k | σ 2 k (cid:107) e k (cid:107) H ≤ δ , then we have
[Equation]
) where ρ ∗ ( I − ˆ ε k H )
= max {| 1 − ˆ ε k λ ∗ min | , | 1 − ˆ ε k λ ∗ max |} .
Generally, we have ρ ∗ ( I − ˆ ε k H ∗ )
≤
ρ ( I − ˆ ε k H ) provided ˆ ε k > 0 , and the optimal rate is ρ ∗ opt :
= κ ∗ − 1 κ ∗ +1 ≤ κ − 1 κ +1 =: ρ opt , where the inequality is strict for almost all u .
Hence, the estimate (15) indicates that the optimal BNGD could have a faster convergence rate than the optimal
GD, especially when κ ∗ is much smaller than κ .
Finally, we discuss the dependence of the effective learning rate ˆ ε k (and by extension, the effective convergence rate (13) or (15)) on ε .
This is in essence a sensitivity analysis on the performance of BNGD with respect to the choice of learning rate.
The explicit dependence of ˆ ε k on ε is quite complex, but we can nevertheless give the following asymptotic estimates.
Proposition 3.5.
Suppose ε a ∈ (0 , 1] , a 0 w T 0 g > 0 , and
|| g || 2 ≥ w T 0 g σ 20 g T Hw 0 , then
(1) When ε is small enough, ε (cid:28) 1 , the effective step size has a same order with ε ,
i
e. there are two positive constants, C 1 , C 2 , independent on ε and k , such that C 1 ≤ ˆ ε k /ε ≤ C 2 .
(2) When ε is large enough, ε (cid:29) 1 , the effective step size has order O ( ε − 1 ) ,
i.e
there are
two
positive
constants
, C , C
, independent on
ε
and k , such that C 1 ≤ ˆ ε k ε ≤ C 2 .
Observe that for ﬁnite k , ˆ ε k is a differentiable function of ε .
Therefore, the above result implies, via the mean value theorem, the existence of some ε 0 >
such that d ˆ ε k
/dε
| ε = ε 0 = 0 .
Consequently, there is at least some small interval of the choice of learning rates ε where the performance of BNGD is insensitive to this choice.
In fact, empirically this is one commonly observed advantage of BNGD over GD, where the former typically allows for a variety of (large) learning rates to be used without adversely affecting performance.
The same is not true for GD, where the convergence rate depends sensitively on the choice of learning rate.
We will see later in Section 4 that although we only have a local insensitivity result above, the interval of this insensitivity is actually quite large in practice.
E XPERIMENTS
Let us ﬁrst summarize our key ﬁndings and insights from the analysis of BNGD on the OLS problem.
A scaling law governs BNGD, where certain conﬁgurations can be deemed equivalent 2.
BNGD converges for any learning rate ε > 0 , provided that ε a ∈ (0 , 1] .
In particular, different learning rates can be used for the BN variables ( a ) compared with the remaining trainable variables ( w ) 3.
There exists intervals of ε for which the performance of BNGD is not sensitive to the choice of ε
In the subsequent sections, we ﬁrst validate numerically these claims on the OLS model, and then show that these insights go beyond the simple OLS model we considered in the theoretical framework.
In fact, much of the uncovered properties are observed in general applications of BNGD in deep learning.
4. E XPERIMENTS ON OLS
Here we test the convergence and stability of BNGD for the OLS model.
Consider a diagonal matrix
H = diag ( h ) where h
= (1 , ..., κ ) is a increasing sequence.
The scaling property (Proposition 3.2) allows us to set the initial value w 0 having same 2 -norm with u , (cid:107) w 0 (cid:107)
= (cid:107) u (cid:107) = 1 .
Of course, one can verify that the scaling property holds strictly in this case.
Figure 1 gives examples of H with different condition numbers κ .
We tested the loss function of
BNGD, compared with the optimal GD (i.e. GD with the optimal step size ε opt ), in a large range of step sizes ε a and ε , and with different initial values of a 0 .
Another quantity we observe is the effective step size ˆ ε k of BN.
The results are encoded by four different colors: whether ˆ ε k is close to the optimal step size ε opt , and whether loss of BNGD is less than the optimal GD.
The results indicate that the optimal convergence rate of BNGD can be better than GD in some conﬁgurations.
This acceleration phenomenon is ascribed to the pseudo-condition number of H ∗ (discard the only zero eigenvalue) being less than κ ( H ) .
This advantage of BNGD is signiﬁcant when the (pseudo)- condition number discrepancy between H and H ∗ is large.
However, if this difference is small, the acceleration is imperceptible.
This is consistent with our analysis in section 3.3.
Another important observation is a region such that ˆ ε is close to ε opt , in other words, BNGD signiﬁcantly extends the range of ‘optimal’ step sizes.
Consequently, we can choose step sizes in BNGD at greater liberty to obtain almost the same or better convergence rate than the optimal GD.
However, the size of this region is inversely dependent on the initial condition a 0 .
Hence, this suggests that small a 0 at ﬁrst steps may improve robustness.
On the other hand, small ε a will weaken the performance of BN.
The phenomenon suggests that improper initialization of the BN parameters weakens the power of BN.
This experience is encountered in practice, such as (Cooijmans et al., 2016), where higher initial values of BN parameter are detrimental to the optimization of RNN models.
a 0 = κ =a 0 = a 0 =0 .a
=0 .a
=1 e −κ = κ = κ =
BNGD GD(opt) l o ss yes
no yes
no loss_BN < loss_GD close to optimal step size
4. E XPERIMENTS ON PRACTICAL DEEP LEARNING PROBLEMS
We conduct experiments on deep learning applied to standard classiﬁcation datasets: MNIST (LeCun et al., 1998), Fashion MNIST (Xiao et al., 2017) and CIFAR-10 (Krizhevsky & Hinton, 2009)
The goal is to explore if the key ﬁndings outlined at the beginning of this section continue to hold for more general settings.
For the MNIST and Fashion MNIST dataset, we use two different networks: (1) a one-layer fully connected network (784 × 10) with softmax mean-square loss; (2) a fourlayer convolution network (Conv-MaxPool-Conv-MaxPool-FC-FC) with ReLU activation function and cross-entropy loss.
For the CIFAR-10 dataset, we use a ﬁve-layer convolution network (ConvMaxPool-Conv-MaxPool-FC-FC-FC).
All the trainable parameters are randomly initialized by the
Glorot scheme (Glorot & Bengio, 2010) before training.
For all three datasets, we use a minibatch size of 100 for computing stochastic gradients.
In the BNGD experiments, batch normalization is performed on all layers, the BN parameters are initialized to transform the input to zero mean/unit variance distributions, and a small regularization parameter (cid:15)
= 1e-3 is added to variance √ σ 2 + (cid:15) to avoid division by zero.
Scaling property Theoretically, the scaling property 3.2 holds for any layer using BN.
However, it may be slightly biased by the regularization parameter (cid:15) .
Here, we test the scaling property in practical settings.
Figure 2 gives the loss of network-(2) (2CNN+2FC) at epoch=1 with different learning rate.
The norm of all weights and biases are rescaled by a common factor η .
We observe that the scaling property remains true for relatively large η .
However, when η is small, the norm of weights are small.
Therefore, the effect of the (cid:15) -regularization in √ σ 2 + (cid:15) becomes signiﬁcant, causing the curves to be shifted.
Stability for large learning rates We use the loss value at the end of the ﬁrst epoch to characterize the performance of BNGD and GD methods.
Although the training of models have generally not converged at this point, it is enough to extract some relative rate information.
Figure 3 shows the loss value of the networks on the three datasets.
It is observed that GD and BNGD with identical learning rates for weights and BN parameters exhibit a maximum allowed learning rate, beyond which the iterations becomes unstable.
On the other hand, BNGD with separate learning rates exhibits a much larger range of stability over learning rate for non-BN parameters, consistent with our theoretical results in Theorem 3.3.
Insensitivity of performance to learning rates Observe that BN accelerates convergence more signiﬁcantly for deep networks, whereas for one-layer networks, the best performance of BNGD and
[Figure]
GD are similar.
Furthermore, in most cases, the range of optimal learning rates in BNGD is quite large, which is in agreement with the OLS analysis (Proposition 3.5).
This phenomenon is potentially crucial for understanding the acceleration of BNGD in deep neural networks.
Heuristically, the “optimal” learning rates of GD in distinct layers (depending on some effective notion of “condition number”) may be vastly different.
Hence, GD with a shared learning rate across all layers may not achieve the best convergence rates for all layers at the same time.
In this case, it is plausible that the acceleration of BNGD is a result of the decreased sensitivity of its convergence rate on the learning rate parameter over a large range of its choice.
[Figure]
C ONCLUSION AND O UTLOOK
In this paper
[Figure]
we adopted a modeling approach to investigate the dynamical properties of batch normalization.
The OLS problem is chosen as a point of reference, because of its simplicity and the availability of convergence results for gradient descent.
Even in such a simple setting, we saw that BNGD exhibits interesting non-trivial behavior, including scaling laws, robust convergence properties, acceleration
[Figure]
as well as the insensitivity of performance to the choice of learning rates.
Although these results are derived only for the OLS model, we show via experiments that these are qualitatively valid for general scenarios encountered in deep learning
[Figure]
and points to a concrete way in uncovering the reasons behind the effectiveness of batch normalization.
Interesting future directions include the extension of the results for the OLS model to more general settings of BNGD, where we believe the scaling law (Proposition 3.2) should play a signiﬁcant role.
In addition, we have not touched upon another empirically observed advantage of batch normalization, which is better generalization errors.
It will be interesting to see how far the current approach takes us in investigating such probabilistic aspects of BNGD.
A P ROOF OF T HEOREMS A. G RADIENTS AND H ESSIAN MATRIX
The objective function in problem (7) has an equivalent form:
[Equation]
where u = H − 1 g
[Equation]
The gradients are:
[Equation]
The Hessian matrix is
[Equation]
19) where
[Equation]
The objective function J ( a, w ) has trivial critical points, { ( a ∗ , w ∗ ) | a ∗ = 0 , w ∗ T g = 0 } .
It is obvious that a ∗ is the minimizer of J ( a, w ∗ ) , but ( a ∗ , w ∗ ) is not a local minimizer of J ( a, w ) unless g = 0 , hence ( a ∗ , w ∗ ) are saddle points of J ( a, w ) .
The Hessian matrix at those saddle points has at least a negative eigenvalue, i.e. the saddle points are strict.
In fact, the eigenvalues at the saddle point ( a ∗ , w ∗ ) are (cid:110) 12 (1 ± (cid:113) 1 + 4 (cid:107)
g (cid:107) 2 w ∗ T Hw ∗ ) , 0 , ..., 0 (cid:111) which contains d − 2 repeated zero, a positive and a negative eigenvalue.
On the other hand, the nontrivial critical points satisﬁes the relations,
[Equation]
22) where the sign of a ∗ depends on the direction of u, w ∗ , i.e. sign ( a ∗ )
= sign ( u T w ∗ ) .
It is easy to check that the nontrivial critical points are global minimizers.
The Hessian matrix at those minimizers is diag (1 , H ∗ / (cid:107) w ∗ (cid:107) 2 ) where the matrix H ∗ is
[Equation]
23) which is positive semi-deﬁnite and has a zero eigenvalue corresponding to the eigenvector u , i.e. H ∗ u = 0 .
The following lemma, similar to the well known Cauchy interlacing theorem, gives an estimate of eigenvalues of H ∗ .
Lemma A.1.
If H is positive deﬁnite and H ∗ is deﬁned as H ∗
=
H − Huu T H u T Hu , then the eigenvalues of H and H ∗ satisfy the following inequalities:
[Equation]
[Equation]
Here λ i ( H ) means the i -th smallest eigenvalue of H .
Proof.
According to the deﬁnition, we have H ∗ u = 0 , and for any x ∈ R d ,
[Equation]
(25) which implies H ∗ is semi-positive deﬁnite, and λ i ( H ∗ ) ≥ λ 1 ( H ∗ ) = 0 .
Furthermore, we have the following equality:
[Equation]
(2) We will prove λ
i ( H ∗ ) ≤ λ i ( H ) for all i , 1 ≤ i ≤ d .
In fact, using the Min-Max Theorem, we have
[Equation]
(3) We will prove λ
i ( H ∗ ) ≥ λ
i −
( H ) for all i , 2 ≤ i ≤ d .
In fact, using the Max-Min Theorem, we have
[Equation]
where we have used the fact that x ⊥ u , (cid:107) x − tu
[Equation] [Equation]
There are several corollaries related to the spectral property of H ∗ .
We ﬁrst give some deﬁnitions.
Since H ∗ is positive semi-deﬁnite, we can deﬁne the H ∗ -seminorm.
Deﬁnition A.2.
The H ∗ -seminorm of a vector x is deﬁned as (cid:107) x (cid:107) H ∗ :=
x T H ∗ x . (cid:107) x (cid:107) H ∗ =
if and only if x is parallel to u .
Deﬁnition A.3.
The pseudo-condition number of H ∗ is deﬁned as κ ∗ ( H ∗ ) :
= λ d ( H ∗ ) λ 2 ( H ∗ ) .
Deﬁnition A.4.
For any real number ε , the pseudo-spectral radius of the matrix I − εH ∗ is deﬁned as
[Equation]
The following corollaries are direct consequences of Lemma A.1, hence we omit the proofs.
Corollary A.5.
The pseudo-condition number of H ∗ is less than or equal to the condition number of H :
[Equation]
(27) where the equality holds up if and only if u ⊥ span { v 1 , v d } , v i is the eigenvector of H corresponding to eigenvalue λ
i ( H ) .
Corollary A.6.
For any vector x ∈ R d and any real number ε , we have (cid:107) ( I − εH ∗ ) x (cid:107) H ∗ ≤ ρ ∗ ( I − εH
[Equation]
) (cid:107)
[Equation]
(cid:107) H ∗ .
Corollary A.7.
For any positive number ε > 0 , we have
[Equation]
(28) where the inequality is strict if u T v i (cid:54)
=
for i = 1 , d .
It is obvious that the inequality in (27) and (28) is strict for almost all u .
A.
S CALING PROPERTY
The dynamical system deﬁned in equation (9)-(10) is completely determined by a set of conﬁgurations { H, u, a 0 , w 0 , ε a , ε } .
It is easy to check the system has the following scaling property:
Lemma A.8 (Scaling property) .
Suppose µ (cid:54) = 0 , γ (cid:54) = 0 , r (cid:54) = 0 , Q T Q
=
I , then
(1) The conﬁgurations { µQ T HQ, γ √ µ Qu, γa 0 , γQw 0 , ε a , ε } and { H, u, a 0 , w 0 , ε a , ε } are equivalent.
The conﬁgurations { H, u, a 0 , w 0 , ε a , ε } and { H, u, a 0 , rw 0 , ε a , r 2 ε } are equivalent.
The scaling property is valid for general loss functions provided batch normalization is used.
Consider a general problem
[Equation]
and its BN version
[Equation]
Then the gradient descent method gives the following iteration,
[Equation]
where ˜ h
= h ( a k w k /σ k ) , and h is the gradient of original problem:
[Equation]
It is easy to check the general BNGD has the following property:
Lemma A.9 (General scaling property) .
Suppose r (cid:54) = 0 , then the conﬁgurations { w 0 , ε, ∗} and { rw 0 , r 2 ε, ∗} are equivalent.
Here the sign * means other parameters.
A. P ROOF OF T HEOREM 3.
Recall the BNGD iterations
[Equation]
The scaling property simplify our analysis by allowing us to set, for example, (cid:107) u (cid:107) =
and (cid:107) w 0 (cid:107) = 1 .
In the rest of this section, we only set (cid:107) u (cid:107) = 1 .
For the step size of a , it is easy to check that a k tends to inﬁnity with ε a > 2 and initial value a 0 = 1 , w 0 = u .
Hence we only consider 0 <
ε a < 2 , which make the iteration of a k bounded by some constant C a .
Lemma A.10 (Boundedness of a k ) .
If the step size 0 < ε a < 2 , then the sequence a k is bounded for any ε > 0 and any initial value ( a 0 , w 0 ) .
Proof.
Deﬁne α k :=
w Tk g σ k , which is bounded by | α k | ≤ √ u T Hu
=:
C , then
[Equation]
Since | 1 − ε a | < 1 , we have | a k +1 | ≤ | a 0 | + 2 C (cid:80) ki =
| 1 − ε a | i ≤ | a
[Equation]
|
[Equation]
− ε
[Equation]
| .
According to the iterations (34), we have
[Equation]
Deﬁne
[Equation] [Equation]
and using the property w
[Equation]
H , and the property of H -norm, we have
[Equation]
Therefore we have the following lemma to make sure the iteration converge:
Lemma A.11.
Let 0 < ε a < 2 .
If there are two positive numbers ε − and ˆ ε + , and the effective step size ˆ ε k satisﬁes
[Equation]
for all k large enough, then the iterations (34) converge to a minimizer.
Proof.
Without loss of generality, we assume
[Equation]
and the inequality (39) is satisﬁed for all k ≥ 0 .
We will prove (cid:107) w k (cid:107) converges and the direction of w k converges to the direction of u .
(1) Since (cid:107) w k (cid:107) is always increasing, we only need to prove it is bounded.
We have,
[Equation]
The inequality in last lines are based on the fact that (cid:107)
He i (cid:107)
≤ λ max (cid:107) e
i (cid:107) 2 H , and | a k | are bounded by a constant C a .
Next, we will prove (cid:80) ∞
i =
q i (cid:107) w i (cid:107) 2 < ∞ , which implies (cid:107) w k (cid:107) are bounded.
According to the estimate (38), we have
[Equation] [Equation]
(45) where 1 − γ + = max
i {| 1 − ˆ ε
+ λ i | 2 } ∈ (0 , 1) .
Using the deﬁnition of q k , we have
[Equation]
Since q k is bounded in [0 , u T Hu ] , summing both side of the inequality, we get the bound of the inﬁnite series (cid:80)
[Equation]
(2) Since (cid:107) w k (cid:107) is bounded, we denote ˆ ε − :=
[Equation]
, and deﬁne
[Equation]
then the inequality (38) implies q k +1 ≤
ρ 2 q k .
As a consequence, q k tends to zero, which implies the direction of w k converges to the direction of u .
(3) The convergence of a k is a consequence of w k converging.
Since a k is bounded, we assume | a k | < ˜ C a √ u T Hu , ˜ C a ≥ 1 , and deﬁne ε 0 :=
[Equation]
a κλ max .
The following lemma gives the convergence for small step size.
Lemma A.12.
If the initial values ( a 0 , w 0 ) satisﬁes a 0 w T 0 g > 0 , and step size satisﬁes ε a ∈ (0 , 1] , ε/ (cid:107) w 0 (cid:107) 2 < ε 0 , then the sequence ( a k , w k ) converges to a global minimizer.
Remark 1: If we set a 0 = 0 , then we have w 1 = w 0 , a 1 = ε a w T 0 g σ 0 , hence a 1 w T 1 g > 0 provided w T
g (cid:54) = 0 .
Remark 2:
For the case of ε a ∈ (1 , 2) , if the initial value satisﬁes an additional condition 0 < | a 0 | ≤ ε a | w T
g | σ 0 , then we have ( a k , w k ) converges to a global minimizer as well.
Proof.
Without loss of generality, we only consider the case of a 0 > 0 , w T 0 g > 0 , (cid:107) w 0 (cid:107) ≥ 1 .
(1) We will prove a k > 0 , w Tk g > 0 for all k .
Denote y k
=
w Tk g , δ = (cid:107) g
[Equation]
4 κ .
On the one hand, if a k > 0 , 0 < y k < 2 δ , then
[Equation]
On the other hand, when a k > 0 , y k > 0 , ε < ε 0 , we have
[Equation]
As a consequence, we have a k > 0 , y k ≥ δ y :
= min { y 0 , δ } for all k by induction.
(2) We will prove the effective step size ˆ ε k satisfying the condition in Lemma A.11.
Since a k is bounded, ε < ε 0 , we have
[Equation]
<
[Equation]
and
[Equation]
which implies w Tk +1 g σ k +1 ≥ w Tk g σ k ≥ w T 0 g σ 0 .
Furthermore, we have a k ≥ min { a 0 , w T 0 g σ 0 } , and there is a positive constant ε − > 0 such that
[Equation]
(3) Employing the Lemma A.11, we conclude that ( a k , w k ) converges to a global minimizer.
Lemma A.13.
If step size satisﬁes ε a ∈ (0 , 1] , ε/ (cid:107) w 0 (cid:107) 2 < ε 0 , then the sequence ( a k , w k ) converges.
Proof.
Thanks to Lemma A.12, we only need to consider the case of a k w T k g ≤ 0 for all k , and we will prove the iteration converges to a saddle point in this case.
Since the case of a k = 0 or w Tk g =is trivial, we assume a k w Tk g < 0 below.
More speciﬁcally , we will prove | a k +1 | < r | a k | for some constant r ∈ (0 , 1) , which implies convergence to a saddle point.
(1) If a k and a k +1 have same sign, hence different sign with w Tk g , then we have | a k +1 | = | 1 − ε a (cid:107) a k | − ε a | w Tk g | /σ k ≤ | 1 − ε a (cid:107) a k | .
If a k and a k +1 have different signs, then we have
[Equation]
(cid:107)
[Equation]
−
[Equation] [Equation]
Consequently, we get
[Equation]
(3) Setting r := max( | 1 − ε a | , 2 εε a κλ max − (1 − ε a )) , we ﬁnish the proof.
To simplify our proofs for Theorem 3.3, we give two lemmas which are obvious but useful.
Lemma A.14.
If positive series f k , h k satisfy f k +1 ≤ rf k + h k , r ∈ (0 , 1) and lim k →∞ h k
=
, then lim
k →∞ f k = 0 .
Proof.
It is obvious, because the series b k deﬁned by b k +1 =
rb k + h k , b 0 > 0 , tends to zeros.
Lemma A.15 (Separation property) .
For δ 0 small enough, the set S :
= { w | y 2 q < δ 0 , (cid:107) w (cid:107) ≥ 1 } is composed by two separated parts: S 1 and S 2 , dist ( S 1 , S 2 ) > 0 , where in the set S 1 one has y 2 <
δ 1 , q > δ 2 , and in S 2 one has q < δ 2 , y 2 > δ 1 for some δ 1 > 0 , δ 2 > 0 .
Here y := w T g, q := u T Hu − ( w T Hu )
2 w T Hw = u T Hu − y 2 w T Hw .
Proof.
The proof is based on H being positive.
The geometric meaning is illustrated in Figure 4.
Corollary A.16.
If lim
[Equation]
[Figure]
[Equation] [Figure] [Equation]
[Figure] [Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
then either lim
[Figure]
[Equation]
or lim k →∞ q k = 0 .
Proof.
Denote y k
=
w Tk g .
According to the separation property (Lemma A.15), we can chose a δ 0 > 0 small enough such that the separated parts of the set S :
= { w | y 2 q < δ 0 , (cid:107) w (cid:107) ≥ 1 } , S 1 and
S 2 , have dist ( S 1 , S 2 ) > 0 .
Because y 2 k q k tends to zero, we have w k belongs to S for k large enough, for instance k > k 1 .
On the other hand, because (cid:107) w k +1 − w k (cid:107) tends to zero, we have (cid:107) w k +1 − w k (cid:107) < dist ( S 1 , S 2 ) for k large enough, for instance k > k 2 .
Then consider k > k 3 :=
max( k 1 , k 2 ) , we have all w k belongs to the same part S 1 or S 2 .
If w k ∈ S 1 , ( q k > δ 2 ), for all k > k 3 , then we have lim k →∞ ( w Tk g ) 2 = 0 .
On the other hand, if w k ∈ S 2 , ( y 2 k > δ 1 ), for all k > k 3 , then we have lim k →∞ q k = 0 .
Theorem A.17.
Let ε a ∈
(0 , 1] and ε > 0 .
The sequence ( a k , w k ) converges for any initial value ( a 0 , w 0 ) .
Proof.
We will prove (cid:107) w k (cid:107) converges, then prove ( a k , w k ) converges as well.
(1) We will prove that (cid:107) w k (cid:107) is bounded and hence converges.
In fact, according to the Lemma A.13, once (cid:107) w k (cid:107) 2 ≥ ε/ε 0 for some k , the rest of the iteration will converge, hence (cid:107) w k (cid:107) is bounded.
(2) We will prove lim k
[Equation]
The convergence of (cid:107) w k (cid:107) implies (cid:80) k
a 2 k q k is summable.
As a consequence,
[Equation]
and lim k →∞ (cid:107) w k +1 − w k (cid:107) = 0.
In fact, we have
[Equation]
Consider the iteration of series | a k − w Tk g/σ k | ,
[Equation]
The constant C in (57) can be chosen as C = ελ max (cid:107) u
[Equation]
H λ min (cid:107) w 0 (cid:107) 2 .
Since (cid:107) a k e k (cid:107) H tends to zero, we can use Lemma A.14 to get lim k →∞ | a k −
w Tk g/σ k | = 0 .
Combine the equation (55), then we have lim k →∞ ( w Tk g ) 2 p k = 0 .
According to the Corollary A.16, we have either lim k
[Equation]
y
[Equation]
k = 0 , or lim
[Equation]
q k = 0 .
In the former case, the iteration of ( a k , w k ) converges to a saddle point.
However, in the latter case, ( a k , w k ) converges to a global minimizer.
In both cases we have ( a k , w k ) converges.
To ﬁnish the proof of Theorem 3.3, we have to demonstrate the special case of ε a = 1 where the set of initial values such that BN iteration converges to saddle points is Lebeguse measure zero.
We leave this demonstration in next section where we consider the case of ε a ≥ 1 .
A. I MPOSSIBILITY OF CONVERGING TO STRICT SADDLE POINTS
In this section, we will prove the set of initial values such that BN iteration converges to saddle points is (Lebeguse) measure zero, as long as ε a ≥ 1 .
The tools in our proof is similar to the analysis of gradient descent on non-convex objectives (Lee et al., 2016; Panageas & Piliouras, 2017).
In addition, we used the real analytic property of the BN loss function (16).
For brevity, here we denote x :=
( a, w ) and let ε a = ε , then the BN iteration can be rewrote as
[Equation]
Lemma A.18.
If A ⊂ T ( R d / { 0 } ) is a measure zero set, then the preimage T − 1 ( A ) is of measure zero as well.
Proof.
Since T is smooth enough, according to Theorem 3 of (Ponomarev, 1987), we only need to prove the Jacobian of T ( x ) is nonzero for almost all x ∈ R d .
In other words, the set { x : det( I − ε ∇ 2 J ( x ))
= 0 } is of measure zero.
This is true because the function det( I − ε ∇ 2 J ( x )) is a real analytic function of x ∈ R d / { 0 } .
(Details of properties of real analytic functions can be found in (Krantz & Parks, 2002) for instance).
Lemma A.19.
Let f : X → R be twice continuously differentiable in an open set X ⊂ R d and x ∗ ∈
X be a stationary point of f .
If ε > 0 , det( I − ε ∇ 2 f ( x ∗ ))
(cid:54) = 0 and the matrix ∇ 2 f ( x ∗ ) has at least a negative eigenvalue, then there exist a neighborhood U of x ∗ such that the following set
B has measure zero,
[Equation]
Proof.
The detailed proof is similar to (Lee et al., 2016; Panageas & Piliouras, 2017).
Deﬁne the transform function as F ( x ) :
= x − ε ∇ f ( x ) .
Since det( I − ε ∇ 2 f ( x ∗ )) (cid:54) = 0 , accorded to the inverse function theorem, there exist a neighborhood U of x ∗ such that T has differentiable inverse.
Hence T is a local C 1 diffeomorphism, which allow us to use the central-stable manifold theorem (Shub, 2013).
The negative eigenvalues of ∇ 2 f ( x ∗ ) indicates λ max ( I − ε ∇ 2 f ( x ∗ )) >and the dimension of the unstable manifold is at least one, which implies the set B is on a lower dimension manifold hence B is of measure zero.
Lemma A.20.
If ε a = ε ≥ 1 , then the set of initial values such that BN iteration converges to saddle points is of Lebeguse measure zero.
Proof.
We will prove this argument using Lemma A.18 and Lemma A.19.
Denote the saddle points set as W := { ( a ∗ , w ∗ ) : a ∗ = 0 , w ∗ T g = 0 } .
The basic point is that the saddle point x ∗ :
=
( a ∗ , w ∗ ) of the BN loss function (16) has eigenvalues (cid:110) 12 (1 ± (cid:113) 1 + 4 (cid:107)
g
(cid:107) 2 w ∗ T Hw ∗ ) , 0 , ..., 0 (cid:111) of the Hessian matrix.
(1) For each saddle point x ∗ :
= ( a ∗ , w ∗ ) of BN loss function, ε ≥ 1 is enough to allow us to use
Lemma
A.19.
Hence there exist a neighborhood U x ∗ of x ∗ such that the following set B x ∗ is of measure zero,
[Equation]
The neighborhoods U x ∗ of all x ∗ ∈ W forms a cover of W , hence, accorded to
Lindel ¨ of’s open cover lemma, there are countable neighborhoods { U i : i = 1 , 2 , ... } cover W , i.e. U :
= ∪ i U i ⊇ W .
As a consequence, the following set A 0 is of measure zero,
[Equation]
(3) Deﬁne A
m
+
1 :=
T −
A m
) =
x ∈ R d :
T x ) ∈
A m } , m ≥ 0 .
According to Lemma A.18, we have all A m and ∪ m A m are of measure zero.
(4) Since each initial value x 0 such that the iteration converges to a saddle point must be contained in some set A m , we ﬁnish the proof.
Combine the results of Lemma A.20, scaling property 3.2 and the convergence theorem A.17, we have the following theorem directly.
Theorem A.21.
If ε a = 1 , ε ≥ 0 , then the BN iteration (9)-(10) converges to global minimizers for almost all initial values.
A. C ONVERGENCE RATE
In the last section, we encountered the following estimate for e k =
[Equation]
We can improve the convergence rate of the above if H ∗ has better spectral property.
This is the content of Proposition 3.4 and the following lemma is enough to prove it.
Lemma A.22.
The following inequality holds,
[Equation]
where δ k :=
λ max ε | a k | σ 2 k (cid:107) e k (cid:107) H .
Proof.
The case of w Tk g = 0 is trivial, hence we assume w Tk g (cid:54) = 0 in the following proof.
Rewrite the iteration on w k as the following equality,
[Equation]
Then we will use the properties of H ∗ -seminorm to prove our argument.
[Equation]
the
H ∗ -seminorm
on the
right hand
of
equation
[Equation]
(2) Estimate
the H ∗ -seminorm on the left hand of equation (
Using the
H -
norm on the iteration of w k , we have
[Equation]
Consequently, we have
[Equation]
(3) Combining (1) and (2), we ﬁnish the proof.
Now
[Equation]
we turn to the convergence of the loss function which can be rewritten as J k = 12 (cid:107) ˜
e k (cid:107) 2 H with ˜
e k = u − a k σ k w k .
There is an useful equality between (cid:107) ˜
e k (cid:107) 2 H and (cid:107) e k (cid:107) 2 H :
[Equation]
Recalling the inequality (57) and the boundedness of a k
[Equation]
we have a constant C 0 such that
[Equation]
which indicates that we can use the convergence of e k to estimate the convergence of the loss value
J k .
In fact we have the following lemma.
Lemma A.23.
If (cid:107) e k (cid:107) H ≤ Cρ k for some constant C and ρ ∈ (0 , 1) , ε a ∈ (0 , 1] , then we have
[Equation]
where γ = max( ρ, 1 − ε a ) , C 1 = | a 0 − w T
g/σ 0 | and C 2 = CC 0 .
Proof.
According to the inequality (71), we have
[Equation]
Put it in the equality (70), then we ﬁnish the proof.
A. E STIMATING THE EFFECTIVE STEP SIZE
Accorded to Lemma A.12, the effective step size ˆ ε k has same order with
[Equation]
0 , ε/ || w
|| <
ε 0 .
In fact, we have
[Equation]
Hence, to prove the Proposition 3.5, we only need to estimate the norm of w k .
proof of Proposition 3.5.
According to the BNGD iteration, we have (see the proof of Lemma A.11)
[Equation]
When
[Equation]
2 < ε 0 ( ε 0 is deﬁned in Lemma A.12), the sequence q k satisﬁes q k +1 ≤ (1 − ˆ ε k λ min ) q k .
Hence the norm of w k is bounded by
[Equation]
for some constant C .
As a consequence,
[Equation]
) (2) When ε is large enough, the increment of the norm (cid:107) w k (cid:107) at the ﬁrst step is large as well.
In fact, we have
[Equation]
Since || g || 2 ≥ w T 0 g σ 20 g T Hw 0 , we have a 1 w T 1 g > a 1 w T 0 g > 0 .
Choose ε to be larger than some value ε 1 such that
[Equation]
< ε 0 , then we can use the argument in (1) on ( a 1 , w 1 ) .
More precisely, there are two constants, C 1 , C 2 , such that
[Equation]
Plugging the equation (78) into it, we have
[Equation]
B M ODIFIED BNGD
Through our analysis, we discovered that a modiﬁcation of the BNGD, which we call MBNGD, becomes much easier to analyze and possesses better convergence properties.
Note that the results in the main paper do not depend on the results in this section.
The modiﬁcation is simply to enforce a k = w Tk g σ k at every iteration, which yields the modiﬁed iterations:
[Equation]
In a sense, one can view the above as a limiting version of BNGD where the BN rescaling variable a is adjusted every step to the optimal value based on the current value of the weights w .
The MBNGD iterations is governed by the variables: H, u, a 0 , w 0 , ε , where the scaling properties (Lemma 3.2, omit the parameter ε a now) remains.
More importantly, we ﬁnd the iteration will converge to a saddle point if and only if it exactly meets the saddle point at a ﬁnite step.
More precisely, we have the convergence theorem:
Theorem B.1 (Convergence for modiﬁed BNGD) .
The iteration sequence w k in equation (81) converges for any initial value w 0 and any step size ε > 0 .
It converges to a global minimizer almost sure, in the sense that the set of initial values such that w k converges to a saddle point is of Lebesgue measure zero.
Furthermore, It converges to a saddle point if and only if w Tk g = 0 for some k .
Particularly, if ε <
[Equation]
= 0 , then w k converges to a global minimizer.
In the following, we assume (cid:107) u (cid:107) = 1 .
B. P ROOF
Lemma B.2.
If w T 0 g (cid:54) =
and
[Equation]
max , then the sequence w k converges to a global minimizer.
Proof.
Similar to the proof of Lemma A.12, but here the effective step size is always nonnegative which is deﬁned as
[Equation]
The inequality (38) immediately gives q k +1 ≤ q k , which implies ( w Tk +1 g ) 2 σ 2 k +
[Equation]
As a consequence, the effective step size has a lower bound
[Equation]
Employing the Lemma A.11, we conclude that w k converges to a global minimizer.
Lemma B.3.
If w Tk g (cid:54) = 0 for all k , then (cid:80) ∞ k =0 | w Tk g | = (cid:80) ∞
k =0 ( w Tk g ) 2 = ∞ .
Proof.
Without loss of generality, we assume (cid:107) w 0 (cid:107) ≥ 1 , denote y k :=
w Tk g and set δ = (cid:107) g (cid:107) 4 κ .
From the iteration of w k , we have
[Equation]
[Equation]
If 0 < | w Tk g | < 2 δ , then we have the inequality:
[Equation]
then
[Equation]
As a consequence, lim k →∞ w Tk g = 0 is not possible unless w Tk g = 0 for some k , which implies the results we want.
Theorem B.4.
The iteration sequence w k in equation (81) converges for any initial value w 0 and any step size ε > 0 .
Furthermore, w k will converge to a global minimizer unless w Tk g = 0 for some k .
Proof.
Obviously, if w Tk g = 0 for some k = k 0 , then w k = w k 0 for all k ≥ k 0 , hence w k converges to w k 0 .
Without losing generality, we consider w Tk g (cid:54) = 0 for all k and (cid:107) w 0 (cid:107) ≥ 1 below.
(1) Firstly, we will prove that (cid:107) w k (cid:107) is bounded and hence converges.
In fact, according to the Lemma B.2, once (cid:107) w k (cid:107) 2 ≥ ε/ε 0 for some k , the rest of the iteration will converge, hence (cid:107) w k (cid:107) is bounded.
(2) Secondly, we will prove w k converges to a vector parallel to u .
Denote y k :=
w Tk g, z k :=
w Tk g σ 2 .
The convergence of (cid:107) w k (cid:107) indicates that (cid:80) ∞ k
=
z 2 k q k is summable, and then (cid:80) ∞ k =
y 2 k q k is summable as well.
Therefore we have
[Equation]
and the above tends to zero, i.e. lim k →∞ (cid:107) w k +1 − w k (cid:107) = 0.
According to the separation property (Lemma A.15), we can chose a δ 0 > 0 small enough such that the separated parts of the set S :
= { w | y 2 q < δ 0 , (cid:107) w (cid:107) ≥ 1 } , S 1 and S 2 , have dist ( S 1 , S 2 ) > 0 .
Because y 2 k q k tends to zero, we have w k belongs to S for k large enough, for instance k > k 1 .
On the other hand, because (cid:107) w k +1 − w k (cid:107) tends to zero, we have (cid:107) w k +1 − w k (cid:107) < dist ( S 1 , S 2 ) for k large enough, for instance k > k 2 .
Then consider k > k 3 :=
max( k 1 , k 2 ) , we have all w k belongs to the same part S 1 or S 2 .
However, Lemma B.3 says (cid:80) ∞ k =
y 2 k = ∞ , hence w k ∈ S 1 ( q k > δ 2 ) for all k > k 3 is not true.
Therefore w k ∈ S 2 ( y 2 k > δ 1 ) for all k > k 3 .
Consequently, we can claim that (cid:80) ∞ k
=
q k is summable and w k converges to a vector parallel to u .
B. E XPERIMENT
Here we test the convergence and stability of MBNGD for OLS model.
Consider the diagonal matrix
H = diag ( h ) , where h = (1 , ..., κ ) is an increasing sequence.
The scaling property allows us to set the initial value w 0 having same 2 -norm with u , (cid:107) w 0 (cid:107) =
(cid:107) u (cid:107) = 1 .
Figure 5 gives an example of a 5-dimensional H with condition number κ = 2000 .
The GD and
MBNGD iteration are executed k = 5000 times where u and w 0 are randomly chosen from the unit sphere.
The values of effective step size, loss (cid:107) e k (cid:107) 2 H and error (cid:107) e k (cid:107) are plotted.
Furthermore, to explore the performance of GD and MBNGD, the mean values over 300 random tests are given.
It is worth to note that, the geometric mean (G-mean) is more reliable than the arithmetic mean (Amean), where the geometric mean of x can be deﬁned as exp( E (ln x )) .
Here the reliability means that the G-mean converges quickly when the number of tests increase, however the A-mean does not converge as quickly.
In this example, the optimal convergence rate of MBNGD is observably better than GD.
This acceleration phenomenon is ascribed to the pseudo-condition number of κ ∗ ( H ∗ ) being less than κ ( H ) .
However, if the difference between (pseudo-)condition number of H and H ∗ is small, the acceleration is imperceptible.
Another important observation is that the BN signiﬁcantly extends the range of ‘optimal’ step size, which is embodied by the effective step size ˆ ε k having a large constant C in ˆ ε
= O ( Cε − 1 ) .
This means we can chose step size in BN at a large interval to get almost same or better convergence rate than that of the best choice for GD.
Figure 6 gives an example of 100-dimension H with condition number κ = 2000 .
Similar results as those in the 5-dimensional case are obtained.
However, the best optimal convergence rate of
MBNGD here has not noticeably improved compared with GD with the optimal learning rate, which is due to the fact that large d decrease the difference between eigenvalues of H and H ∗ .
Additional tests indicate that:
) larger dimensions leads to larger intervals of ‘optimal’ step size, (Figure
7) (2) the effect of condition number on
the
‘optimal’ interval
is small (Figure