Surprising Instabilities in Training Deep Networks and a Theoretical Analysis
Abstract
We discover restrained numerical instabilities in current training practices of deep networks with stochastic gradient descent (SGD).
We show numerical error (on the order of the smallest floating point bit) induced from floating point arithmetic in training deep nets can be amplified significantly and result in significant test accuracy variance, comparable to the test accuracy variance due to stochasticity in SGD.
We show how this is likely traced to instabilities of the optimization dynamics that are restrained, i.e., localized over iterations and regions of the weight tensor space.
We do this by presenting a theoretical framework using numerical analysis of partial differential equations (PDE), and analyzing the gradient descent PDE of a simplified convolutional neural network (CNN).
We show that it is stable only under certain conditions on the learning rate and weight decay.
We reproduce the localized instabilities in the PDE for the simplified network, which arise when the conditions are violated.
Introduction
Deep learning is now standard practice in a number of application areas including computer vision and natural language processing.
However, the practice has out-paced the theory.
Theoretical advances in deep learning could improve practice, and could also speed the adoption of the technology in e g., a number of safety critical application areas.
In this paper we discover a phenomena of restrained instabilities in current deep learning training optimization.
Numerical instabilities are present but are restrained so that there is not full global divergence, but these restrained instabilities still cause tiny numerical finite precision errors to amplify, which leads to significant variance in the test accuracy.
We introduce a theoretical framework using tools from numerical partial differential equations (PDE) to provide insight into the phenomena and analytically show how such restrained instabilities can arise as a function of learning rate and weight decay for a simplified CNN model.
This study is a step toward a theory for principally choosing learning rates in deep network training from the perspective of numerical conditioning.
In deep learning practice, it is well known that large learning rates can cause divergence and small learning rates can be slow and cause the optimization to be stuck at high training error [1, 13].
Learning rates and schedules are often chosen heuristically to address this trade-off.
On the other hand, in numerical PDEs, there is a mature theory for choosing the learning rate [38] or step size for discrete schemes for solving PDEs.
There are known conditions on the step size (e.g., the CourantFriedrichs-Lewy (CFL) condition [9]) to ensure that the resulting scheme is stable and converges.
These methods would thus be natural to study learning rate schemes for training deep networks with
SGD, which in the continuum limit is a PDE.
However, to the best of our knowledge, numerical
PDEs have not been applied to this domain.
This might be for two reasons.
First, the complicated structure of deep networks with many layers and non-linearities makes it difficult to apply these analytic techniques.
Second, the stochastic element of SGD complicates applying these techniques, which primarily do not consider stochastic elements.
While we do not yet claim to overcome these issues, we do show the relevance of PDE analysis in analyzing training of deep networks with SGD, which gives promise for further developing this area.
Our particular contributions are: 1.
We found convincing evidence of and thus discovered the presence of restrained numerical instabilities in standard training practices for deep neural networks.
We show that this results in inherent floating points arithmetic errors having divergence effects on the training optimization that are even comparable to the divergence due to stochastic variability in SGD.
We present and study a simplified model, which allows us to exploit numerical PDE analysis, in order to offer an explanation on how these instabilities might arise.
These insights suggest to us that the learning rate should be chosen adaptively over localized spacetime regions of the weight tensor space in order to ensure stability while not compromising speed.
While we understand this is a complex task, without a simple solution, our study motivates the need for investigating such challenging strategies.
Note there are a number of heuristically driven methods for adaptive learning rates (e.g., [12, 43, 21, 2, 27]), but they do not address numerical stability.
Related Work
As stated earlier, we are not aware of related work in applying numerical PDE analysis to study stability and choice of learning rates of deep network training 2 .
There has been work on studying convergence of SGD and learning rates (e.g., [24, 29, 37]), but they are often based on convexity and Lipschitz assumptions that could be difficult to apply to deep networks.
For deep networks, there is work on global stability of training in relation to curvature of the loss, e.g., [8].
[8] is more focused on generalization rather than numerical stability and does not address restrained instabilites.
More broadly, although not directly related to our work, there has been a number of recent works in connecting PDEs with neural networks for various end goals (see [40, 19, 4] for detailed surveys).
For example, applying neural networks to solve PDEs [34, 7, 26, 20, 16], and interpreting the forward pass of neural networks as approximations to PDEs [32, 15, 39].
Since SGD can be interpreted as a discretization of a PDE, PDEs also play a role in developing optimization schemes for neural networks [5, 6, 41, 28, 23, 36, 35].
Despite various methods [11, 18, 25] proposed to reduce variance of final accuracy induced by the stochastic gradient, we find quite surprisingly that even if all random seeds in optimization are fixed, variance induced by epsilon-small numerical noise (perturbing only the last floating point bit) and hence floating point arithmetic is nevertheless greater or similar to variance across independent trials (see Table 1).
This observation potentially indicates that numerical stability is a critical but long-ignored factor affecting neural network optimization.
There has been work on training deep networks using limited numerical precision (e.g., half precision) [10, 14].
The purpose of these works is to introduce schemes to retain accuracy, which is well known to be reduced significantly when lowering precision to these levels.
Different from this literature, we show an unknown fact: errors just in the last bit of floating point representations result in significant divergence.
This is to provide evidence of numerical instability resulting from learning rate choices.
Background: PDEs, Discretization, and Numerical Stability
We start by providing background on analyzing stability of linear PDE discretizations.
This methodology will be applied to neural networks in later sections as a PDE is the continuum limit of SGD.
We illustrate concepts using a basic PDE, i.e., the heat equation (used to model a diffusion process):
[Equation]
where u : [0 , T ] × R → R , ∂ t u denotes the partial with respect to time t , ∂ xx u is the second derivative with respect to the spatial dimension x , and u 0 : R → R is the initial condition.
To solve this numerically, one discretizes the equation, using finite difference approximations.
A standard discretization is through ∂ xx u ( t, x ) =
u
[Equation]
+ O (
[Equation]
2 ) and
[Equation]
t u
[Equation]
+ O (∆ t ) , where ∆ x is the spatial increment and ∆ t is the step size, which gives the following update scheme:
[Equation]
where n denotes the iteration number, u n ( x ) is the approximation of u at t = n ∆ t , and ϵ denotes the error of the approximation (i.e., due to discretization and finite precision arithmetic).
A key question is whether u n remains bounded as n → ∞ , i.e., numerically stable .
It is typically easier to understand stability in the frequency domain, which is referred to as Von Neumann analysis [38, 31].
Computing the spatial Discrete Fourier Transform (DFT) yields:
[Equation]
[Equation] [Equation] (3) where the hat denotes DFT, ω denotes frequency, and A ( ω ) = 1 − ∆ t (∆ x ) 2 [1 − cos( ω ∆ x )] is the amplifier function.
Note that u n is stable if and only if | A ( ω ) | < 1 .
Notice that if | A ( ω ) | < 1 , u n converges and errors ϵ n are attenuated over iterations.
If | A ( ω ) | ≥ 1 , u n diverges and the errors ϵ n are amplified.
The case, | A ( ω ) | < 1 for all ω , implies the conditions that ∆ t > 0 and ∆ t < 12 (∆ x ) 2 .
This is known as the CFL condition.
Notice the restriction on the time step (learning rate) for numerical stability.
If the discretization error of the operator on the right hand side of the PDE is less than O (∆ x ) , stability also implies convergence to the PDE solution as ∆ t → 0 [38, 31].
See Figure 1 for simulation of the discretization scheme for the heat PDE.
The instability causes blowup of the solution globally.
For standard optimization of CNNs, we will show rather that the instability is restrained (localized in time and space), which nevertheless causes divergence.
Empirical Evidence of Instability in Training Deep Networks
We discover and provide empirical evidence of restrained instabilities in current deep learning training practice.
To this end, we show that optimization paths in current practice of training convolutional neural networks (CNNs) can diverge significantly due to the smallest errors from finite precision arithmetic.
We show that the divergence can be eliminated with learning rate choice.
4.1 A Perturbed SGD by Introducing the Smallest Floating Point Perturbations
We introduce a modified version of SGD that introduces perturbations of the original SGD update on the order of the smallest possible machine representable perturbation, modeling noise on the order of error due
to finite precision floating point arithmetic:
[Equation]
where η > 0 is the learning rate, g t is the stochastic gradient or the momentum vector if momentum is used, and k > 1 is an odd integer.
Notice that mathematically, both updates are exactly equivalent (division by k cancels the multiplication by k ), however considering floating point arithmetic, these may not be equivalent.
A floating point number is represented as significand × base exponent
[Equation]
where typically base 2 (binary) is used.
Thus, the modified update introduces a potential perturbation in the last significant bit of ηg t , hence a relative difference of the right hand sides on the order of 2 − s according to IEEE standards on floating point computation.
In other words, RelativeError ( x, y ) := | x − y | / | x | < 2 − ( s − 1) where x = fl ( ηg t ) ,
y =
fl ( η/k ) × fl ( kg t ) , and fl ( x ) is the floating point representation of x .
Note that this is the same order of relative error between ηg t and fl ( ηg t ) , the inherent machine error due to floating point representation.
The perturbation induces the smallest possible perturbation of ηg t representable by the machine.
Thus, perturbed SGD is a way to show the effects of machine error.
Notice that when k is a power of two, as floating point numbers are typically stored in binary form, the division just subtracts the exponent and the multiplication adds to the exponent so both updates are equivalent and no perturbation is introduced.
However, choosing k odd can create changes at the last significant bit, with each choice of k yielding a different change.
4.2 Perturbed SGD on Common CNNs and Demonstration of Restrained Instabilities
In SGD, there are several sources of randomness, i.e., random initialization, random shuffled batches/selection, random data augmentation, and there are also sources of non-determinism in implementations of the deep learning frameworks.
To ensure the variance merely originated from the floating point perturbation, in the following experiments, we eliminate all randomness by fixing the initialization, the random seed for batch selection, and the non-determinism flag.
Appendix B empircally validates that all seeds are fixed.
We use Pytorch using default 32 bit floating point precision (similar conclusions for 64 bit hold).
Perturbed SGD with ReLU : Our first experiment uses the ResNet-56 architecture [17], which we train on CIFAR-10 [22] using perturbed SGD.
We use the standard parameters for training this network [17]: lr=0.1, batch size = 128, weight decay = 5e-4, momentum=0.9 .
The standard step decay learning rate schedule is used: the learning rate is divided by 10 every 40 epochs for a total of 200 epochs.
We report the final test accuracies for 6 different values of k , including k =(standard SGD), and compute the standard deviation of the accuracies (STD).
Table 1 (left) shows the results.
This results in test accuracy variability over different k .
We repeat this experiment over different seeds for the stochastic batch selection (shown in the next columns) fixing the initialization.
The standard deviation for each seed over k is on average 0 .
To illustrate the significance of this value, we compare to the variability due to batch selection on the right of Table 1.
We empirically estimate the relative fluctuation of the gradient estimate from batch selection, which is 26 . 72 .
This is large compared to 2 − 23 for the floating point perturbation, yet this smallest fluctuation that is machine representable (and models finite precision error) results in the about the same accuracy variance as batch selection.
Thus, finite precision error, inherent in the system, surprisingly results in about the same test accuracy variance as stochastic batch selection.
Perturbed SGD with Swish : We first conjectured that the variance due to floating point errors could be caused by the ReLU activation, the activation standard in ResNets.
This because the gradient of the ReLU is discontinuous at zero.
Hence small positive values (e.g., 1e-6) would give a gradient ofand small negative values (e.g., -1e-6) would give a gradient of zero.
Hence the ReLU gradient is sensitive to numerical fluctuation around zero.
Note common deep learning implementations choose a value at zero, among 0, 0.5 or 1; but this has the same sensitivity.
Therefore, we adopted the Swish activation function (proposed in [30]), Swish ( x ) :=
x/ [1 + exp ( − βx )]
( β > 0 , as β → ∞ Swish approaches ReLU), which is is an approximation of the ReLU that has a continuous gradient at zero.
Recent work has shown the Swish function improves accuracy [30] as well.
Thus, we repeated the previous experiment (under the same settings) replacing the ReLU with the Swish activation.
Results are shown in Appendix, and show similar results as the previous experiment with ReLU: the
S EED
STD k =93.
k =93. k =93.
k =93. k =93.
k =92.
STD 0.
P ERTURBATION
R ELATIVE
T EST A CC M ETHOD
G RAD .
E RROR
S TD
F LOATING P T 2 −0.
B ATCH S ELECTION 26.
Other Architectures/Datasets : To verify that this phenomenon is not specific to the choice of network architecture or dataset, we repeated the same experiment for a different network (VGG [33]) and a different dataset (Fashion-MNIST [42]).
We summarize the results in the Appendix, which confirms that the phenomena is still present across different networks and datasets.
Divergence in Optimization Paths : We plot the difference in network weights (using the measure described next) between the original SGD weights, θ i ( i is the index for location in the vector), and the perturbed SGD weights, θ ′ i over epochs and show that the errors are amplified, suggesting instability.
We use the average relative L1 absolute difference between weights: RelL1 ( θ, θ ′ ) :
=
N (cid:80) i | θ i − θ ′
i | / ( | θ i | + | θ ′ i | ) , where N is the number of network parameters.
This measure is used as the numerical error is multiplicative and relative to the weight.
Note that RelL1 is bounded by 2.
Results are shown for multiple networks in Figure 2 (left).
The difference rapidly grows, indicating errors are amplified, and then the growth is restrained at larger epochs.
Although the errors are restrained, the initial error growth is enough to result in different test accuracies.
Evidence Divergence is Due to Instability : We provide evidence that this divergence phenomenon is due to an instability by showing that decreasing the learning rate to a small enough value can eliminate the divergence, even initially.
We experiment with various fixed learning rates.
We use
Resnet56 under the same settings as before.
For smaller learning rates, more iterations would have to be run to move an equivalent amount for a larger rate.
We thus choose the iterations for each run such that the learning rate times the number of iterations is constant.
The result is shown in Figure(right), where the axis is scaled as mentioned.
When the learning rate is chosen small enough, the floating point errors are attenuated over iterations, indicating the process is stable (note it is not stuck at a local minima - see Appendix C). Above this level, the errors are amplified.
PDE Stability Analysis of a Simplified CNN
In this section, we use numerical PDE tools introduced in Section 3 to analyze a one-layer CNN to show analytically how the instability phenomenon described in the previous section can arise.
To do this, we start with a continuum model of the network, derive the analytical formula for the gradient descent, which is a PDE, discretize it, which serves as a model for the optimization algorithm, and analyze the stability of the discretization through Von-Neumann analysis.
5.1 Network, Loss and Gradient Flow PDE
Let I : R 2 → R be an input image to the network, K : R 2 → R be a convolution kernel (we assume
I and K to be of finite support), r : R → R be the Swish activation, and s : R → R is the sigmoid function.
We consider the following CNN, which is the simplest version of VGG:
[Equation]
This network consists of a convolution layer, an activation layer, a global pooling layer and a sigmoid layer.
As in deep learning packages, ∗ will denote the cross-correlation (which we call convolution).
The bias is not included in the network as it does not impact our analysis nor conclusions.
We assume a dataset that consists of just a single image I and its label y ∈ { 0 , 1 } .
This is enough to demonstrate the instability phenomena.
We consider the following regularized cross-entropy loss:
[Equation]
where
[Equation]
[Equation]
ℓ denotes cross-entropy, the second term is regularization ( L 2 norm squared of the kernel), and α >is a parameter (weight decay).
Conclusions are not are restricted to cross-entropy, but it makes some expressions simpler.
The gradient descent PDE is given by (see the Appendix for proof):
Theorem 5.1 (Gradient Descent PDE) .
The gradient descent PDE with respect to the loss (16) is
[Equation]
where r ′ denotes the derivative of the activation, and t parametrizes the evolution.
If we maintain the constraint that K is finite support so that K is zero outside [ − w/ 2 , w/ 2] 2 then the constrained gradient descent is
[Equation]
where W is a windowing function ( 1 inside [ − w/ 2 , w/ 2] 2 and zero outside).
5.2 Stability Analysis of Optimization
We now analyze the stability of the standard discretization of the gradient descent PDEs.
Note that the
PDEs are non-linear, and in order to perform the stability analysis, we study linearizations of the PDE around the switching regime of the activation, i.e., K = 0 .
As we will see in the next sub-section, away from the transitioning regime, a similar analysis applies.
We first study the linearization of (26), and then we adapt the analysis to (27).
The linearization of the PDE results in the following (see Appendix for the derivation):
Theorem 5.2 (Linearized PDE) .
The linearization of the PDE (27) around K = 0 is given by
[Equation]
where
[Equation]
a is considered constant with respect to K , ¯ I denotes the integral (sum) of the values of I (assumed finite support), and β > 0 is the parameter of the Swish activation, r ( x ) :=
[Equation]
In Appendix D, we also consider a non-constant linear model of K for a .
This leads to the similar conclusions as the constant model, and so we present the constant model for simplicity.
We discretize the linearization, which results in a discrete optimization algorithm, equivalent to the algorithm employed in standard deep learning packages.
Using forward Euler discretization gives
[Equation]
where n denotes the iteration number, and ∆ t denotes the step size (learning rate in SGD).
To derive the stability conditions, we compute the spatial DFT of (35) (see the Appendix D for details):
Theorem 5.3 (DFT of Discretization) .
The DFT of the discretization (35) of the linearized PDE is
[Equation]
where ˆ K n denotes the DFT of K n , and sinc denotes the sinc function.
In the case that w → ∞ (the window support becomes large), the DFT of K can be written in terms of the amplifier A as ˆ K
[Equation]
where A
[Equation]
In order for the updates to be a stable process, the magnitude of the amplifier should be less than one, i.e., | A | < 1 .
This results in the following stability criteria (see the Appendix D for a derivation):
Theorem 5.4 (Stability Conditions) .
The discretization of the linearized PDE (26) whose DFT is given by (36) is stable (in the case w → ∞ , i.e., the window gets large) if and only if the following conditions on the weight decay α and the step size (learning rate) ∆ t are met:
[Equation]
Note the sign of a = ˆ y − y can be either positive or negative; in the case a > 0 , the second condition is automatically met (since β > 0 ).
However in the case that a < 0 , the weight decay must be chosen large enough to be stable.
The first condition shows that the weight decay and the step size satisfy a relationship if the optimization is to be stable.
Note that the conditions also depend on the structure of the network, the current state of the network (i.e., through dependence of a ), and the frequency content of the input.
This is different than current deep learning practice, where the weight decay is typically chosen constant through the evolution and does not adapt with the state of the network, and the step size is typically chosen empirically without dependence on the weight decay.
5.3 Empirical Validation of Stability Bounds for Linear PDE
Figure 3 empirically validates the existence of upper and lower bounds on α .
We chose the learning rate ∆ t = 1e-8, a = − 0 .
5 , β = 1 , initialize the weights K ( 32 × 32 ) according to a normal distribution (mean 0, variance 1), and I is a 256 × 256 checkerboard pattern filled with 1 and -1.
The above stability conditions were under the condition that the windowing function, W , had infinite support.
Thus, the evolution could cause the initial kernel’s (finite) support to grow over iterations, which is not representative of fixed, finite support kernels used in deep network practice.
In the case that W is finite and fixed support, it is not possible to analytically find a simple multiplicative amplifier factor as in (37) since the convolution with the sinc function in (36) does not separate from ˆ K n .
In the case that ˆ K is a constant (i.e., K is a delta function or support of size 1), ( ˆ K n | ˆ I | 2 ) ∗ sinc (cid:0) w 2 · (cid:1) =
ˆ K n ( | ˆ I | 2 ∗ sinc (cid:0) w 2 · (cid:1) ) , in which case one can write the amplifier function as
[Equation]
which has the effect that the upper bound for α increases and the lower bound decreases in (13), but maintains existence of upper and lower bounds.
We conjecture this is also true when K has support larger than 1 , and the bounds on α converge to (13) as the support of the kernel is increased.
We provide empirical evidence.
We find the lower and upper bounds for α by running the PDE and noting the values of α when the scheme first becomes [Table] unstable.
We compare it to the bounds in (13).
We use the same settings as the previous experiment.
See results in Table 2, which verifies the bounds approach the infinite support case.
In this section we analyzed the linearized gradient descent PDE.
Momentum is often used.
See
Appendix E for analysis of the momentum case, which leads to similar conclusions as this section.
5.4 Restrained Instabilities in the Non-linear PDE
We now explain how the linear analysis around K = 0 in the previous sections applies to analyzing full non-linear PDE.
In particular, we show that the non-linear PDE is also not stable unless similar conditions on the weight decay and step size are met.
However, the non-linear PDE can transition to regimes that have more generous stability conditions where it could be stable and the PDE can move back and forth between stable and unstable regimes, resulting in restrained instabilities.
Approximation of Non-Linear PDE by Linear PDEs and Stability Analysis : The non-linear
PDE could move between three different regimes: when the activation is “activated” ( r ′ ( x ) = 1 ), “not activated” ( r ′ ( x ) = 0 ), and “transitioning” ( r ′ ( x ) is a non-constant linear function).
In the transitioning regime, K ≈ 0 , in which case the linear analysis of the previous section applies, and in this regime, there are conditions on the weight decay and step size for stability.
In the not-activated regime, K ∗ I is negative and away from zero, the non-linear PDE is driven only by the regularization term, which is stable when α < 2∆ t .
This is typically true in current deep learning practice where weight decay is chosen on the order of 1e-4 and the learning rate is on order of < 1e-1.
In the activated regime, K ∗ I is positive and away from zero and the non-linear PDE can be approximated by a linear
PDE (see Appendix D.5), which gives the stability condition:
α < 2∆ t − ¯ I 2 8 , similar to the stability condition in the transitioning regime.
Formation of Restrained Instabilities : Note that different regions in the kernel domain could each be in different regimes and therefore governed by different linear PDEs discussed earlier.
See Table 3.
Such regions could then switch between different regimes throughout the evolution of the PDE, and thus the evolution could go between stable and unstable regimes.
For example, in the transitioning regime, the instability quickly drives the magnitude of the kernel up (if the stability conditions are not met) and thus the region to a possibly stable (e.g., activated) regime.
Data-driven terms could then kick the kernel back into the (unstable) transitioning regime, and this switching between regimes could happen indefinitely.
As a result instabilities occur, but can be “restrained” by the activation.
The optimization nevertheless amplifies and accumulates small noise in the transitioning regime.
Empirical Validation : We now empirically demonstrate that restrained instabilites are present and error amplification occurs in the (non-linear) gradient descent PDE of the one layer CNN even when the dataset consists of a single image.
We choose α = 20 , β = 1 , y = 1 and the same input image I and kernel K initialization as specified in Section 5.3.
Figure 4 (left) shows the loss function over
Activated
Transitioning Not Activated [Table] iterations for various learning rate ( ∆ t ) choices, and the right plot shows the evolution of the L relative difference between weights of SGD ( k = 1 ) and perturbed SGD ( k = 3 ) as discussed in
Section 4.
From the previous discussion and (13), we know that it is necessary that ∆ t < 0 .
1 in order for all regimes to be stable.
This is confirmed by the plot on the left: when ∆ t > 0 . 1 , the loss blows up.
When ∆ t < 0 .
03 , which is the empirically determined threshold for the transitioning region, all regimes are stable, and the kernel converges as shown in the plot.
When 0 .
03 < ∆ t ≤ 0 .
1 , the
PDE transitions between the (unstable) transitioning regime and the stable one (activated), which introduces oscillations.
This is consistent with our theory outlined previously.
On the plot on the right of Figure 4, we show divergence caused by the floating perturbations introduced in Section 4 for various ∆ t .
When 0 .
03 < ∆ t ≤ 0 .
1 (in the restrained instability regime), the error builds up quickly and then levels off.
When ∆ t < 0 .
03 , errors are not amplified.
This is consistent with the behavior of multi-layer networks in Section 4 validating our theory.
Remarks on Deeper Networks :
This section formulated theory on how the divergence phenomena for practical deep network optimization reported in Section 4 could arise.
We found that even the simplest (one-layer) CNN exhibits this phenomena through analytical techniques.
In multi-layer networks, there are more possibilities for such restrained instabilities to arise in multiple layers and multiple kernels of the network.
Such instabilities may only arise in some kernels/layers.
The analysis in this section is relevant to any one layer of such networks by treating the part of the network before the layer as input; within a small time period, one can treat this an a nearly constant input to the layer.
Conclusion
We discovered restrained numerical instabilities in standard training procedures of deep networks.
In particular, we showed that epsilon-small errors inherent in floating point arithmetic can amplify and lead to divergence in final test accuracies, comparable to stochastic batch selection.
Given such variations are comparable to accuracy gains reported in many papers, such instabilities could inflate or even suppress such gains, and further investigation is needed.
Using a simplified CNN, which is amenable to linear analysis, we employed PDEs to explain how these instabilites might arise.
We derived CFL conditions, which imposed conditions on the learning rate and weight decay.
This study provided a step in principally choosing learning rates for stability.
We showed that numerical PDEs have promise to shed insight.
The analysis showed how restrained instabilities can arise even in a simplified CNN, which suggests the same phenomena exists in larger networks.
Although empirically we showed how restrained instabilites could be eliminated with globally small learning rates, such instabilities are localized to space-time regions of the tensor space, suggesting in practice the need to develop adaptive learning rate schemes tailored to have small rates in these local regions to ensure numerical stability, while maintaining speed. Acknowledgements This research was supported in part by Army Research Labs (ARL) W911NF-22-1-0267 and Raytheon Technologies Research Center.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See remark 1 in page 2.
(c) Did you discuss any potential negative societal impacts of your work?
[No] , general theory same impacts of deep learning technology in general (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See section(b)
Did you include complete proofs of all theoretical results?
[Yes] Yes, appendix 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplementary (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See section 5.3 and section 5.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Extended Discussion
What is the relation between Section 3 and Section 4?
Section 3 presents theory on how finite precision errors can be attenuated or amplified based on satisfying or not the CFL condition (condition on the learning rate) through PDE tools.
Note the continuum limit of SGD as the learning result goes to zero is a PDE.
As such and for ease of illustration, a simple PDE is examined to illustrate the key ideas of the PDE framework.
Section 4 presents that such amplification of finite precision errors is present in deep network optimization, and shows that this likely results from not satisfying the CFL condition (learning rate is too high) - last experiment of the section.
What is the relation between Section 4 and Section 5?
Section 5 presents a detailed theoretical study on how the amplification / attenuation of finite precision errors from Section 4 can arise by applying methodology from Section 3.
The study reduces down restrained instabilities to its most basic manifestation in simple one-layer CNN trained on just a single image, and shows how they arise in this CNN.
In particular, when the learning rate is chosen in a specific range, restrained instabilites (oscillating between stable and unstable regimes) arise and finite precision errors accumulate.
For small enough learning rates, the instabilites disappear and finite errors are attenuated.
This matches the behavior of the large networks in Section 4, suggesting the theory applies.
What is the significance of this work?
We have taken a step in developing theory for principally choosing and designing learning rate schemes from the perspective of numerical stability, showing in particular the relevance of numerical PDE theory, which has not been explored before.
The study also suggests the exploration of adaptive optimization schemes that are constructed to ensure numerical stability.
Whether eliminating restrained instabilities in training results in better performance (e.g., generalization, training time, robustness, reduced variance, etc) is an open question for future exploration.
It may be well the case that such stabilities are helpful in training, but this requires further investigation and our theory provides a starting point for answering that question.
The work also discovered the phenomena of restrained instabilites and error amplification, and significant test accuracy variance resulting from that phenomena.
Whether eliminating the instability reduces stochastic test variance from batch selection / initialization is an open question.
Moreover, given that there are several papers in literature where performance reported is on the order of the variance reported in this paper, our result may challenge the significance of those results, and further investigation is needed.
B Empirical Validation: All Seeds in Experiments of Section 4 are Fixed
We verify that all random seeds in SGD and the deep learning framework (e.g., initialization, batch selection, CUDA seeds, etc) have been fixed in the experiments in Section 4.2. This would indicate that the only sources of variance across trials is due to the introduced floating point perturbation. To verify that, we run experiments for multiple trials when k = 1 .
Test accuracy over epochs from each trial is shown in the following Table 4.
This confirms that the dynamic trajectories of weights strictly coincide, indicating that all seeds have been fixed.
[Table]
To further strengthen this point, we provide additional results on k ′ =
k × 2 n in Table 5.
Since base 2 (binary) is used in floating number, multiplying or dividing by the power of 2 will not introduce floating point arithmetic perturbation, therefore for any integer k , results on k ′ and k will have the same trajectories.
This is another piece of evidence showing that the instability comes ONLY from floating point error.
We also provided code in the supplementary (main4.py) for verification.
k= k= k= k= k= k= k= k= k= k= k= k=
Trial
[Table]
[Table]
Trial
[Table]
[Table]
[Table]
[Table] [Table] C Supplementary Experimental Results for Section We provide supplementary experimental results for Section 4.
The bold titles correspond to the section in Section 4.2 for which the supplementary experiment belongs.
Section 4.2, Perturbed SGD with Swish : Table 6 shows detailed results of the variability due to the floating point perturbation in comparison to batch selection with Swish activation for Section 4.
Perturbed SGD with Swish .
As noted in Section 4.2, the Swish activation also results in significant variance due to floating point arithmetic.
[Table]
Section 4.2, Other Architectures/Datasets : Table 7 verifies that the variability due to the floating point perturbation generally exists across different network architectures and datasets.
Here we repeated the same experiment for a different network (VGG16) and a different dataset (FashionMNIST).
The Swish activation is used.
Six different floating point noises are used and six different seeds are used as in Table 6.
Average standard deviations for test accuracy variation for both the floating point perturbations and batch selection.
The test accuracy variance for floating point noise is greater or similar to the stochastic variation due to batch selection.
Note the average test accuracy and the average standard deviation over floating point perturbations over k are reported.
The variance of SGD due to batch selection is also reported.
[Table]
Section 4.2, Evidence of Divergence Due to Instability : We verify that the optimization for the stable learning rate (3.125e-6) does not cause fluctuation for various k in perturbed SGD due to being stuck at a local minima.
To this end, we show the test accuracy plot for k = 1 , 3 in Figurecorresponding to the experiment in the right side of Figure 2 in the main paper.
Note that the accuracy is increasing, indicating the optimization is not stuck at a local minima.
D Derivation for Gradient Descent PDE and Details of Stability Analysis D.1 Derivation of Gradient PDE (Theorem 5.1)
Let I : R 2 → R be an input image to the network, K : R 2 → R be a convolution kernel (we assume
I and K to be of finite support), r : R → R be the Swish activation, and s : R → R is the sigmoid function.
We consider the following CNN, which is the simplest version of VGG:
[Equation]
We consider the following regularized cross-entropy loss:
[Equation]
where ˆ y = f
( I ) (16) where ℓ denotes the cross-entropy, the second term denotes the regularization ( L 2 norm squared of the kernel, K ), and α > 0 and the second term is weight regularization.
Define
[Equation]
Computing the variation of g ( I ) with respect to δK yields
[Equation]
[Equation]
Therefore,
[Equation]
Therefore,
[Equation]
Note s ( x ) = 1 1+ e − ( x − b ) is the sigmoid (with bias b ), and thus s ′ ( x ) = e
[Equation]
Therefore, s ′ ( g ( I ))
=
f ( I )[1 − f ( I )]
= ˆ y (1 − ˆ y ) .
Therefore,
[Equation]
For cross-entropy, we have ∂ℓ∂ ˆ y =
− y
ˆ y + 1 − y 1 − ˆ y
and thus ˆ y (1 − ˆ y )
∂ℓ∂ ˆ y =
ˆ y − y .
Therefore,
[Equation]
25) where ˆ y =
f ( I ) .
The gradient descent PDE with respect to the loss (16) is
[Equation]
26) where r ′ denotes the derivative of the activation, and t parametrizes the evolution.
If we maintain the constraint that K is finite support so that K is zero outside [ − w/ 2 , w/ 2] 2 then the constrained gradient descent is
[Equation]
(27) where W is a windowing function ( 1 inside [ − w/ 2 , w/ 2] 2 and zero outside).
D.2 Derivation of Linearization of the Gradient PDE (Theorem 5.2)
We assume that r is the Swish function and β > 0 is the parameter of the Swish activation:
[Equation]
We can show that the first derivative and the second derivative of Siwsh function are:
[Equation]
We assume K ∗ I is near zero, so that we could express r ′ ( x ) using Taylor expansion:
[Equation]
where
[Equation]
In this case,
[Equation]
[Equation]
where ¯ I is the input sum.
Assuming ˆ y − y
=: a is approximately constant, the PDE becomes
[Equation]
D.3 Derivation of DFT and CFL Conditions for Lineaized PDE (Theorem 5.3, 5.4)
Consider the forward Euler scheme of (34):
[Equation]
[Equation]
where n denotes the iteration number, and ∆ t denotes the step size (learning rate in SGD).
The DFT of the discretization (35) of the linearized PDE is
[Equation]
where ˆ K n denotes the DFT of K n , and sinc denotes the sinc function.
In the case that w → ∞ (the window support becomes large), the DFT of K can be written in terms of the amplifier A as
[Equation]
To be stable, | A | < 1 .
Provided that α is large enough (when a < 0 ), stability can be achieved with the condition:
[Equation]
38) with α > − 12 aβ max ω | ˆ I ( ω ) | 2 (when a < 0 ), and in this case, we could choose
[Equation]
There are two condition it has to be satisfied to be stable.
First, as ∆ t is positive, the denominator of the right side should be positive.
That yields the lower bound for α
[Equation]
[Equation] Then ∆ t has to satisfy the CFL condition (39) so that the system could be stable, which yields the upper bound for α :
[Equation]
D.4 Non-constant Linearization of a = ˆ y − y and Stability Analysis (Section 5.2)
We also consider a non-constant linear model of K for a , so we linearize a = ˆ y − y around K = 0 :
[Equation]
where the subscript on g indicates the dependence on the given kernel.
The gradient descent PDE following from the linearization above then becomes:
[Equation]
Considering only linear terms and ignoring the constant (w.r.t K ) terms, this becomes:
[Equation]
Using forward Euler and computing the Fourier transform yields:
[Equation]
where
[Equation]
where δ is a Dirac delta function.
Note s (0) − y can be either positive or negative, so similar to the previous case where a = ∂ℓ∂ ˆ y s ′ was assumed constant, the process can be unstable without regularization.
We approximate δ with a sinc function (Fourier transform of a rectangular pulse, which is the case if the constant is defined on just a finite support), i.e.,
[Equation]
We can write A as
[Equation]
[Equation]
The sign of 12 [ s (0) − y ]
β | ˆ I ( ω ) | 2 + 14 s ′ (0)
| ˆ I (0) | 2 δ ( ω ) can be either positive or negative, therefore, in the case of no regularization, the process can be unstable.
This analysis shows that treating a = ˆ y − y as a more general non-constant linear function leads to similar CFL conditions and conclusions as the constant case examined in the main paper.
D.5 Linearization of the PDE in
the “Activated” Regime (Section 5.4)
Suppose that K ∗ I is positive and away from zero, then r ′ ( K ∗ I )
= 1 .
The non-linear PDE (26) reduces to
[Equation]
where a = ˆ y − y .
Linearizing this as in Theorem 4.2 gives a = s (0) − y + 0 .
5 s ′ (0)
¯ K ¯ I , gives the
PDE:
[Equation]
For stability analysis, we can ignore the constant with respect to K term.
One can show that in the
DFT domain, the update of K t is given by
[Equation]
where the amplifier factor is
[Equation]
For stability, | A | < 1 , which implies the following conditions:
[Equation]
and α < min
[Equation]
E Stability Analysis of Nesterov Momentum (Section 5.2)
We now extend the stability analysis from the gradient descent of the 1-layer CNN loss function to consider Nesterov momentum, which is widely used in deep neural network training.
As we shall see, we arrive at similar conclusions (CFL conditions that result in upper and lower bounds on α ) as the gradient PDE.
We start with the continuum PDE, discretize it - resulting in Nesterov’s scheme, and then analyze the stability.
The continuum equivalent PDE for the optimization of a loss function with momentum is given by (see [3]):
[Equation]
where ∂ tt denotes the second derivative in time, d > 0 is a constant that denotes the damping coefficient, and u denotes the evolving variable of optimization.
We may use a semi-implicit Euler style discretization of the above PDE that results in classic two-part Nesterov recursion (see [3]).
Theorem E.1 (Semi-Implicit Scheme for Momentum PDE) .
The semi-implicit scheme for momentum
PDE in (50) can be expressed as
[Equation]
[Equation]
(52) where V t is the partial update, and d denotes the damping coefficient.
Proof.
We start with discretizing PDE with momentum (50) using a fully explicit scheme.
Specifically, we use a central difference approximations for both time derivatives gives a second order discretization in time:
[Equation]
which leads to the following update:
[Equation]
where K n = K ( n ∆ t ) , and ∆ K n = K n − K n − 1 .
To obtain an update which more closely resembles the classic two-part Nesterov recursion, we use a semi-implicit discretization.
That is, we replace the explicit discretization ∇ K L ( K n ) with a “predicted estimate” (cid:92) ∇ K L ( K n ) .
This estimate is obtained by applying the same discretization but evaluating ∇ K L at the partial update
V n = K n + 2 − d ∆ t 2+ d ∆ t ∆ K n , which is a “look-ahead” location.
Using this strategy yields this two-step update as specified in the theorem.
For more details see [3].
To perform the stability analysis, we study linearizations of the PDE by using a similar procedure as in the stability analysis for gradient descent PDE presented in Section 5.3.
The linearization of ∇ K L ( V n ) is the same as in the linearized PDE (34), given by
[Equation]
[Equation]
We now analyze the stability of the linearized equation using Von-Neumann analysis.
To derive the stability conditions, we compute the spatial Discrete Fourier Transform (DFT) of the semi-implicit scheme (51) and solve for the gradient amplifier:
Theorem E.2.
The DFT of the semi-implicit scheme given in (51) - (52) of the linearized momentum
PDE is given by
[Equation]
where the gradient amplifier z is defined as
[Equation]
In order for the overall combined update updates to be a stable process, the gradient amplifier has to satisfy the following condition (for detailed proof, see[3]):
[Equation]
This results in the following stability criteria:
Theorem E.3.
The semi-implicit scheme of the linearized momentum PDE (51) whose DFT is given by (56) is stable (in the case w → ∞ ) if and only if the following conditions on the weight decay α and the step size ∆ t are met: [Equation]
[Equation]
We conduct a similar experiment as in section 5.2 to empirically validate this when the kernel is restricted to be finite support, i.e., the gradient is windowed.
We empirically find the lower and upper bounds for α using the same method and compare it to the bounds in (60) and (61).
We choose the learning rate = 1e-4, a = − 0 .
5 , β = 1 , and damping d = 2 − 2 × 0 .
9 ∆ t (1+0 . 9) so that the momentum coefficient 2 − d ∆ t 2+ d ∆ t = 0 .
9 is the common choice in deep learning training.
Note that the choice of damping d has no impact on the stability bounds.
The input data is the same as the experiment in Section 5.2.
See
Table 8, which verifies there are bounds on α for stability when using momentum as is the case for the gradient descent.
[Table]
F Stability Condition for Multi-layer Networks
In the main paper, we analyzed a simplified one-layer CNN through numerical PDE tools, and extending the analysis to deeper networks with precise conditions as we showed for the 1-layer case may be more difficult.
However, in this supplement, we show that our theory can makes predictions on multi-layer networks used in practice.
Although this result is for highly regularized networks not employed in practice, it does show the validity of our theory.
We obtain a bound on the stable step size when the weight decay α is large.
To do this, we generalize our linearized PDE stability analysis in Section 5.2 to general (multi-layer) networks.
In this case, the linearized gradient descent PDE is
[Equation]
We can perform Von-Neumann analysis by linearizing ∇ K ℓ (ˆ y, y ) and keeping the non-homogeneous component.
If we discretize and compute the DFT, we can get an update scheme of the form ˆ
K t +1 ( ω ) =
A ( ω ) ˆ K t ( ω ) , where the amplifier for the multi-layer network is given by
[Equation]
(63) where ˆ F ( ω ) is the DFT of non-homogeneous part representing the linearization of the gradient of the cross-entropy loss, ℓ .
Noting that the discretization is stable when | A | < 1 results in the following restriction on the time-step (learning rate):
[Equation]
Note that while ˆ F may be complicated to compute exactly analytically for complex multi-layer networks.
We can nevertheless obtain a prediction on the time-step in the case when α is large, i.e., when α is large (highly regularized), the step size restriction approaches:
[Equation]
We verify the preceding bound empirically on various standard multi-layer networks, by choosing a a regularization level and finding empirically the learning rate when the weight updates move from stable to unstable.
We compare this to the bound predicted by (65).
Results are shown in Figure 6.
They show that as the regularization increases, the empirically determined maximum learning rate approaches the theoretical bound above, validating (64).
The ratio of the empirically determined maximum stable learning rate to the theoretical maximum stable learning rate ∆ t = 2 /α estimated by our analysis.
As the L2 regularization (weight decay) increases, the ratio approaches 1, indicating that our bound becomes more accurate.