O N N ONDETERMINISM AND I NSTABILITY IN N EURAL N ETWORK O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Optimization nondeterminism causes uncertainty when improving neural networks, with small changes in performance difﬁcult to discern from run-to-run variability.
While uncertainty can be reduced by training multiple copies of a model with different random seeds, doing so is time-consuming, costly, and makes reproducibility challenging.
Despite this, little attention has been paid towards establishing an understanding of this problem.
In this work, we establish an experimental protocol for understanding the effect of optimization nondeterminism on model diversity, which allows us to study the independent effects of a variety of sources of nondeterminism.
Surprisingly, we ﬁnd that each source of nondeterminism all have similar effects on multiple measures of model diversity.
To explain this intriguing fact, we examine and identify the instability of model training, when taken as an end-to-end procedure, as the key determinant.
We show that even one-bit changes in initial model parameters result in models that converge to vastly different values.
Last, we demonstrate that recent methods in accelerated model ensembling hold promise for reducing the effects of instability on run-to-run variability.
I NTRODUCTION
Consider this common scenario: you have a baseline “current best” model, and are trying to improve it.
Now, one of your experiments has produced a model whose metrics are slightly better than the baseline.
Yet you have your reservations — how do you know the improvement is “real”, and not due to random ﬂuctuations that create run-to-run variability?
Similarly, consider performing hyperparameter optimization, in which there are many possible values for a set of hyperparameters, and you ﬁnd minor differences in performance between them.
How do you pick the best hyperparameters, and how can you be sure that you’ve actually picked wisely?
In both scenarios, the standard practice is to perform multiple independent training runs of your model to understand its variability.
While this does indeed help address the problem, it can be extremely wasteful, increasing the time required for effective research, using more computing power, and making reproducibility more difﬁcult, while still leaving some uncertainty.
Ultimately, the source of this problem is the nondeterminism in optimizing models — randomized components of model training that cause each training run to produce different models with their own performance characteristics.
Nondeterminism itself occurs due to many factors: while the most salient source is the random initialization of parameters, other sources exist, including random shufﬂing of training data, per-example stochasticity of data augmentation, any explicit random operations (e.g. dropout (Srivastava et al., 2014)), asynchronous model training (Recht et al., 2011), and even nondeterminism in low-level libraries such as cuDNN (Chetlur et al., 2014), which are present to improve throughput on hardware accelerators.
Despite the clear impact nondeterminism has on the efﬁcacy of modeling, relatively little attention has been paid towards understanding its mechanisms, even in the classical supervised setting.
In this work, we establish an experimental protocol for analyzing the impact of nondeterminism in model training, allowing us to quantify the independent effect of each source of nondeterminism.
In doing so, we make a surprising discovery: each source has nearly the same effect on the variability of ﬁnal model performance.
Further, we ﬁnd each source produces models of similar diversity, as measured by correlations between model predictions, functional changes in model performance while ensembling, and state-of-the-art methods of model similarity (Kornblith et al., 2019).
To emphasize one particularly interesting result: nondeterminism in low-level libraries like cuDNN can matter just as much with respect to model diversity and variability as varying the entire network initialization.
We explain this mystery by demonstrating that it can be attributed to an inherent numerical instability in optimizing neural networks — when training with SGD-like approaches, we show that small changes to initial parameters result in large changes to ﬁnal parameter values.
In fact, the instabilities in the optimization process are extreme: changing a single weight by the smallest possible amount within machine precision ( ∼ 6 ∗ 10 − 11 ) produces nearly as much variability as all other sources combined .
Therefore, any source of nondeterminism that has any effect at all on model weights is doomed to inherit at least this level of variability.
Last, we present promising results in reducing the effects of instability.
While we ﬁnd that many approaches result in no apparent change, we demonstrate that methods for accelerated model ensembling actually do reduce the variability of trained models without an increase in model training time, providing the ﬁrst encouraging signs for tractability of the problem.
R ELATED W ORK N ONDETERMINISM .
Relatively little prior work has studied the effects of nondeterminism on model optimization.
Within reinforcement learning, nondeterminism is recognized as a signiﬁcant barrier to reproducibility and evaluating progress in the ﬁeld (Nagarajan et al., 2018;
Henderson et al., 2018; Islam et al., 2017; Machado et al., 2018).
In the setting of supervised learning, though, the focus of this work, the problem is much less studied.
Madhyastha & Jain (2019) aggregate all sources of nondeterminism together into a single random seed and analyze the variability of model attention and accuracy as a function of it across various NLP datasets.
They also propose a method for reducing this variability (see Sec.
A for details of our reproduction attempt).
More common in the ﬁeld, results across multiple random seeds are reported (see Erhan et al.
for a particularly extensive example)
, but
the precise nature of nondeterminism’s inﬂuence on variability goes unstudied.
I NSTABILITY .
We use the term “stability” to refer to numerical stability, in which a stable algorithm is one for which the ﬁnal output (converged model) does not vary much as the input (initial parameters) are changed.
Historically, the term “stability” has been used both in learning theory (Bousquet &
Elisseeff, 2002), in reference to vanishing and exploding gradients (Haber & Ruthotto, 2017), and in the adversarial robustness community for a particular form of training (Zheng et al., 2016).
N ONDETERMINISM
Many sources of nondeterminism exist when optimizing neural networks, each of which can affect the variability and performance of trained models.
We begin with a very brief overview:
P ARAMETER I NITIALIZATION .
When training a model, parameters without preset values are initialized randomly according to a given distribution, e.g. a Gaussian with mean 0 and variance determined by the number of input connections to the layer (Glorot & Bengio, 2010;
He et al., 2015).
D ATA S HUFFLING .
In stochastic gradient descent (SGD), the overall gradient is approximated by the gradient on a random subset of examples.
Most commonly, this is implemented by shufﬂing the training data, after which the data is iterated through in order.
Shufﬂing may happen either once, before training, or in between each epoch of training, the variant we use in this work.
D ATA A UGMENTATION .
A very common practice, data augmentation refers to randomly altering each training example to artiﬁcially expand the training dataset.
For example, in the case of images, it is common to randomly ﬂip an image, which encourages invariance to left/right orientation.
S TOCHASTIC R EGULARIZATION .
Some forms of regularization, such as Dropout (Srivastava et al., 2014), take the form of stochastic operations in a model during training.
Dropout is the most common instance of this type of regularization, with a variety of others also in relatively common use, such as
DropConnect (Wan et al., 2013), variational dropout (Gal & Ghahramani, 2016), and variable length backpropagation through time (Merity et al., 2017), among many others.
L OW - LEVEL O PERATIONS .
An underlooked source of nondeterminism, the very libraries that many deep learning frameworks are built on, such as cuDNN (Chetlur et al., 2014) often are run nondeterministically for performance reasons.
This nondeterminism is small in magnitude — in one test we performed this caused a difference of roughly 0 .
In the case of cuDNN, the library we test with, it is possible to disable nondeterministic behavior, incurring a speed penalty typically on the order of ∼ 15% .
However, unlike the other sources of nondeterminism, it is not possible to “seed” this nondeterminism; it is only possible to turn it on or off, but not control its nondeterministic behavior.
3. P ROTOCOL FOR T ESTING E FFECTS OF N ONDETERMINISM D IVERSITY IN P ERFORMANCE .
Our protocol for testing the effects of sources of nondeterminism is based on properly controlling for each source.
In general, suppose there are N sources of nondeterminism, with source i controlled by a seed S i .
To test the effect of source i , we keep all values S j , j (cid:54) =
i set to a constant, and vary S i with R different values, where R is the number of independent training runs performed.
For sources of nondeterminism which cannot be effectively seeded, such as cuDNN, we indicate one of these values as the deterministic value, which it must be set to when varying the other sources of nondeterminism.
For example, suppose that we wish to study three sources of nondeterminism, denoting S 1 the seed for random parameter initialization, S 2 for training data shufﬂing, and S 3 for cuDNN, where S 3 =is the deterministic value for cuDNN.
To test the effect of random parameter initialization, with a budget of R = 30 training runs, then we set S 3 to the deterministic value of 1 , S 2 to an arbitrary constant (also 1 for simplicity), and test 30 different values of S 1 .
All together, this corresponds to training models for ( S 1 , S 2 , S 3 ) ∈ { ( i, 1 , 1) } 30 i =1 , producing a set of 30 models.
To look at variability according to a particular evaluation metric ( e.g. cross-entropy or accuracy for classiﬁcation), we calculate the standard deviation (across all R = 30 models) of the metric.
Note that it is also possible to test the effects of several sources of nondeterminism in tandem this way.
For example, to test all sources of nondeterminism together, the set of models can be changed to ( S 1 , S 2 , S 3 ) ∈ { ( i, i, 0) } Ri =1 .
D IVERSITY IN R EPRESENTATION .
Beyond looking at diversity of test set generalization, though, it is worth examining how different the representations of trained models actually are — even though the diversity in performance might be similar between models trained with different types of nondeterminism, it might be the case that one type of nondeterminism produces models that have learned largely similar concepts, with the variance in generalization due to other factors.
In order to rigorously examine these, we consider four distinct analyses on the functional behavior of models:
The ﬁrst and most straightforward metric we consider is the average disagreement between pairs of models, where higher disagreement corresponds to higher levels of diversity. In contrast to our other metrics, this considers only the argmax of a model’s predictions, which makes it both the most limited and the most interpretable of the group. This metric has also been used recently to compare network similarity in the context of network ensembles (Fort et al., 2019). Second, we consider the average correlation between the predictions of two models, i.e. the expectation (across pairs of models from the same nondeterminism source), of the correlation of predictions, calculated across examples and classes.
For example, for a classiﬁcation problem, the predicted logits from each of R models are ﬂattened into vectors of length N ∗ C (with N the number of test examples and C the number of classes), and we calculate the mean correlation coefﬁcient of the predictions across all (cid:0) R 2 (cid:1) pairs of models.
We use Spearman’s ρ for the correlation coefﬁcient, but note that others such as Pearson’s r are possible and yield similar conclusions.
For this metric, a lower score indicates a more diverse set of models.
The third analysis we perform examines the change in performance from ensembling two models from the same source of nondeterminism.
The intuition is as follows:
If a pair of models are completely redundant, then ensembling them would result in no change in performance.
However, if models actually learn different representations, then we expect an improvement from ensembling, with a greater improvement the greater the diversity in a set of models.
Denoting by f ( S i ) some particular evaluation metric f calculated on the predictions of model S i , this change is equivalent to:
[Equation]
[Table] [Figure] Last, for a more detailed view of learned representations within a network, we consider a stateof-the-art method for measuring the similarity of learned neural network representations, centered kernel alignment (CKA) (Kornblith et al., 2019), which has been used to analyze models trained with different random initializations, widths, and even entirely different architectures.
We use the linear version of CKA, which Kornblith et al.
(2019) found performed similarly to an RBF kernel.
3. E XPERIMENTS IN IMAGE CLASSIFICATION
We begin our study of nondeterminism with the fundamental task of image classiﬁcation.
We test our protocol using CIFAR-10 (Krizhevsky et al., 2009), a 10-way classiﬁcation dataset with 50 ,training images of resolution 32 × 32 pixels and 10 , 000 for testing.
In these initial experiments, we use a 14-layer ResNet model (He et al., 2016), trained with a cosine learning rate decay (Loshchilov & Hutter, 2016) for 500 epochs with a maximum learning rate of .
40 , three epochs of learning rate warmup, a batch size of 512, momentum of 0 . 9 , and weight decay of 5 · 10 − 4 , obtaining a baseline accuracy of 90 . 0% .
Data augmentation consists of random crops and horizontal ﬂips.
All experiments were done on two NVIDIA Tesla V100 GPUs with pytorch (Paszke et al., 2019).
We show the results of our protocol in this setting in Table 1.
What we ﬁnd is surprising — while there are slight differences, each source of nondeterminism has very similar effects on the variability of ﬁnal trained models, measured across both accuracy and cross-entropy.
In fact, random parameter initialization, arguably the most salient form of nondeterminism, does not stand out based on any of these metrics.
Even when combining multiple sources of nondeterminism, there’s remarkably little difference — nothing more than one or two standard deviations compared to any individual source of nondeterminism.
In short, each source of nondeterminism has roughly the same effect on the variability in ﬁnal model performance, with no combination of nondeterminism sources a factor of even two more important than any other. [Table]
[Figure] Turning toward representational diversity on a per-layer level [Figure] average CKA values across 6 representative layers are shown in Fig.
[Figure] done for pairwise combinations of 25 models (due to the computational requirements of CKA).
Consistent with other analyses, CKA reveals that while there are some differences in representational similarity between models trained with different sources of nondeterminism, especially the output of the ﬁrst residual block, by and large these differences are small, easily dwarfed in size by differences across layers
[Figure]
3. E XPERIMENTS IN LANGUAGE MODELING
Here we show that this phenomenon is not unique to image classiﬁcation, applying the same experimental protocol to language modeling.
For these experiments, we employ a small quasirecurrent neural network (QRNN)
(Bradbury et al., 2016) on Penn Treebank (Marcus et al., 1993), using the publicly available code of Merity et al.
This baseline model uses a 256-dimensional word embedding size, 512 hidden units per layer, and 3 layers of recurrent units, obtaining a perplexity (PPL) of 75.49 on the Penn Treebank test set.
For this task, two sources of nondeterminism are relevant: the random parameter initialization, and multiple stochastic operations including variations of dropout and variable length backpropagation through time, which share a common seed.
To measure diversity in performance, PPL is the most widely-accepted metric, and for diversity in representation we focus on only two metrics (pairwise disagreement and beneﬁts from ensembling) because CKA was not designed for variable-length input and common computing libraries (Virtanen et al., 2020) are not efﬁcient enough to calculate O ( R 2 ) correlation coefﬁcients with such large inputs.
We show results in Table 2, where we ﬁnd even less difference in variability between the two sources of nondeterminism considered across all three diversity metrics, showing that the phenomenon isn’t unique to image classiﬁcation or ResNets.
3. N ONDETERMINISM THROUGHOUT TRAINING
One hypothesis for the cause of this phenomenon is that models are particularly sensitive to changes in the initial phase of learning, which recent work has shown to be the case in other contexts (Achille et al., 2019; Frankle et al., 2020).
With our experimental protocol, this is straightforward to test: If this were the case, then training networks identically for the ﬁrst N epochs and only then introducing these sources of nondeterminism would result in a signiﬁcantly lower diversity in trained models, measured across any metric of choice, e.g. variance in cross-entropy or accuracy.
Furthermore, by varying N , we can actually determine when in training each source of nondeterminism has its effect (for sources that vary over the course of training, i.e. not random parameter initialization). [Table]
[Table] We do this experiment for the ResNet-14 model on CIFAR-10 in Fig.
2, where we ﬁnd that it is not only the beginning of training that is sensitive to changes in nondeterminism, refuting this hypothesis about the onset of the effects of nondeterminism.
Instead, the effect of nondeterminism is nearly as high when enabling it even after 50 epochs (10% of the way through training), and we see only a gradual reduction in ﬁnal model variability as the onset of nondeterminism is moved later and later.
I NSTABILITY
Why does each source of nondeterminism have a similar effect on model diversity?
We answer this question by ﬁnding the smallest possible change that produces as diverse a range of models, revealing the instability in optimizing neural networks.
4. I NSTABILITY AND NONDETERMINISM
To demonstrate, consider again a ResNet-14 trained on CIFAR-10, with a cross-entropy loss on the test set of 0 .
3519 and accuracy of 90 . 0% .
Now consider another model trained in an identical fashion, with exactly equal settings for all sources of nondeterminism, but one extremely small change: we randomly pick a single weight in the ﬁrst layer and change its value by the smallest amount possible in a 32-bit ﬂoating point representation, i.e. an addition or subtraction of a single bit in the least-signiﬁcant digit.
As an example, in one model this changed a value from approximately − 0 .
to − 0 .
0066514313 , a difference on the order of 5 · 10 − 10 .
What happens when we optimize this model, which differs from the original only by a miniscule amount, in a single weight, itself within a single layer?
By the end of the ﬁrst epoch of training, still in a phase where the learning rate is warming up, the new model already differs in accuracy by 0 .
18% compared to the original ( 25 . 74% vs 25 .
With one more epoch of training the difference is a larger 2 .
33% ( 33 . 45% vs 31 .
12% ), and by the end of the third epoch, as the learning rate warmup has just ended, the difference is a staggering 10 .
27% vs 30 .
Finally, at the end of training, the differences shrink again, with the new model converged to an accuracy of 90 .
12% and a cross-entropy of 0 . 34335 .
When viewing the optimization process end-to-end, with the initial parameters as the input and a given performance metric as the output, this implies a condition number (cid:107) δf (cid:107) (cid:107) δx (cid:107) of at least 1 . 8 · 10 7 for cross-entropy and 2 . 6 · 10 8 for accuracy.
To identify whether instability is the primary reason for why each source of nondeterminism appears equally important, we can follow our testing protocol from Sec.
3 — this time, our source of nondeterminism is randomly picking a different weight to change in each model training run, then either incrementing or decrementing it to the next available ﬂoating-point value.
We show the results in Table 3 for image classiﬁcation on CIFAR-10 ( c.f.
Table 1 for comparison) and Table 4 for language modeling on Penn Treebank ( c.f.
Table 2), where we ﬁnd that even this miniscule change produces roughly as much variability in model performance as every other source of nondeterminism.
From this, it is easy to see why every other source of nondeterminism has similar effects — so long as a nondeterminism source produces any change in model weights, whether by changing the input slightly, altering the gradient in some way, or any other effect, it will incur at least as much model variability as caused by instability of model optimization.
[Table]
4. I NSTABILITY IS DUE TO DEPTH
Instability is due to network depth, with any network greater than a single layer exhibiting instability.
Theoretically, a linear model, optimized with a cross-entropy and with an appropriate learning rate schedule, should always converge to a global minimum due to convexity.
In practice, we ﬁnd an even stronger property: when we modify the initial weights by a single bit, beyond simply converging to the same weights, the entire optimization trajectory stays close to that of an unperturbed model, never differing by more than a vanishingly small amount.
At convergence, a set of linear models trained in this way with only single random bit changes had a ﬁnal accuracy SD of 0 ( i.e. no changes in any test set predictions) and cross-entropy SD of ∼ 3 · 10 − 8 , far below that of any deeper model.
In contrast, instability occurred as soon as a single hidden layer was added, with an accuracy SD of 0.28 and cross-entropy SD of 0.0055 for a model with a fully-connected hidden layer, and an accuracy SD of 0.14 and cross-entropy SD of 0.0024 when the hidden layer is convolutional, both a factor of 10 , 000 greater than the linear model. See Appendix Table 8 for full details and Appendix Sec. C for a visualization of the effects of instability during training.
A P ATH F ORWARD
Here we identify and demonstrate a family of approaches that holds promise for reducing the variability caused by nondeterminism and instability.
For learnings on approaches which did not reduce variability, we direct the reader to Appendix Sec. A.
As previously mentioned, the standard practice for mitigating run-to-run variability is to train multiple independent copies of a model, gaining a more robust estimate for performance by considering an aggregate measure of a metric of interest ( e.g. the mean). An alternative approach is ensembling, which shares the intuition of multiple independent training runs, but differs in that the predictions themselves are averaged, after which only the performance of the ensembled model itself is compared. By itself, ensembling does not substantially reduce the burden caused by nondeterminism and instability, since it requires the same computation as training multiple copies of models. Despite this, there’s a way out: recent works investigating accelerated ways of constructing ensembles, in which only one training run is needed (Huang et al., 2017; Garipov et al., 2018; Wen et al., 2020).
While these approaches typically underperform ensembles composed out of truly independent models, the nature of their accelerated training provides compelling motivation for reducing variability.
The approach we focus on is Snapshot Ensembles (Huang et al., 2017), which creates an ensemble of models during the training of a single model.
It does this via usage of a cyclic learning rate schedule, where the members of the ensemble are drawn from instances during training when the learning rate goes to 0 (before rising again due to the cyclic learning rate).
We compare a snapshot ensemble approach with 5 cycles in its learning rate ( i.e. model snapshots are taken after every 100 epochs of training) to ordinary ensembling, in which multiple models are trained entirely from scratch.
Results are shown in Table 5, where we focus on CIFAR-10 with all sources of nondeterminism enabled.
The snapshot ensemble we identiﬁed, despite training only a single model, had variability in [Table] accuracy and cross-entropy comparable to an ensemble of 2, with other metrics comparable to those of a vanilla ensemble with even more members.
By reducing the variability in model performance without increasing training time, this identiﬁes accelerated model ensembling as a promising approach for further reducing the trained model variability and mitigating the effects of instability.
G ENERALIZATION E XPERIMENTS
In this section we detail additional experiments showing our nondeterminism and instability claims generalize to other settings.
For computational reasons (each result requires training the same model end-to-end R = 100 times), we restrict our attention to modestly-sized datasets and models.
On CIFAR-10, in addition to the ResNet-14 experimented with in Sec. 3 and Sec. 4 and the linear and single hidden layer models from Sec. 4.2, we experiment with 6-, 10-, and 18-layer ResNet variants, VGG-11 (Simonyan & Zisserman, 2014), and a 50%-capacity ShufﬂeNetv2 (Ma et al., 2018), allowing further exploration of the effect of model size and architecture on nondeterminism and instability.
As shown in Table 6, the observations around instability and its relationship to nondeterminism generally hold for these architectures, with a close correspondence between the magnitude of effects for a random bit change and each of the four metrics considered.
We have also validated the beneﬁts of accelerated ensembling via 5-checkpoint snapshot ensembles with all nondeterminism sources enabled for each of the models, where we ﬁnd large and consistent reductions in run-to-run variability for pairwise disagreement, correlation, and changes from ensembling. For variability in performance (Accuracy and Cross-Entropy SD), we found increasing beneﬁts as model size increased, where only for the smallest ResNet-6 was there no beneﬁt.
We also perform experiments on MNIST (LeCun et al., 1998), allowing us to test whether our observations hold for tasks with very high accuracy — 99 .
14% for our baseline model.
As before, we ﬁnd similar effects of nondeterminism for parameter initialization and all nondeterminism sources, including a comparable effect (albeit smaller) from a single random bit change, highlighting that the instability of training extends even to datasets where the goal is simpler and model performance is higher.
Of note, though, is the relative smaller effect of a single bit change on pairwise metrics of diversity, further suggesting that the magnitude of instability might be at least partially related to the interplay of model architecture, capacity, and degree of overﬁtting.
Snapshot ensembles, however, did not result in improvements on MNIST.
While this is not particularly important in practice (since models on MNIST are fast to train), it is an interesting result, which we hypothesize is also related to the degree of overﬁtting (similar to ResNet-6 on CIFAR-10).
C ONCLUSION
In this work, we have shown two surprising facts: First, though conventional wisdom is that run-to-run variability in model performance is primarily determined by random parameter initialization, many sources of nondeterminism actually result in similar levels of variability.
Second, a key driver of this phenomenon is the instability of model optimization, in which changes on the order of 10 − 10 in a single weight at initialization can have as much effect as reinitializing all weights to completely random values.
Model instability itself is a property of the depth of a model, with any model with more than single layer suffering from it when optimized with SGD.
Beyond identifying and explaining this property of neural network optimization, we have also identiﬁed a promising direction for reducing the variability in model performance and representation without incurring the cost of training multiple copies of a model: accelerated model ensembling.
We hope that our work sheds light on a complex phenomenon that affects all deep learning researchers.
Ultimately, we also hope that, aided by this understanding the ﬁeld is able to improve the stability of model training and accelerate progress in deep learning for all.
A A PPROACHES THAT DON ’ T REDUCE INSTABILITY
In the process of ﬁnding an approach that reduces run-to-run variability of models (Sec. 5) [Table] we experimented with many approaches which all failed to make a dent in improving variability and stability.
For the beneﬁt of the ﬁeld, here we provide our experiences with these approaches which did not succeed in improving stability, despite the intuitive arguments for why they might help.
L EARNING RATE AND DURATION OF TRAINING .
Noticing that the effects of nondeterminism seemed to accumulate during the course of training (Fig. 2), it seemed reasonable that varying the learning rate or duration of training might have an effect.
However, varying the duration of training from anywhere between 50 and 2 , 000 epochs on CIFAR-10 all produced models with a similar variance in performance as the results in the rest of this work (which used 500 epochs), even though the absolute performance differed by up to ∼ 2% . We show these results in Table 7. In general, increasing the number of epochs or changing the learning rate did not change the variability in performance (Accuracy SD; Cross-Entropy SD) much, with only a very slight increase in variability as the number of epochs grew to extremely large values ( i.e. 2,000 epochs). There were slightly larger changes in pairwise representation-based metrics, where training longer again increased run-to-run variability.
However, none of these attempts actually reduced variability; they only served to potentially make it larger.
As part of these experiments, we also veriﬁed the instability behavior with only 200 epochs of training and the effectiveness of accelerated ensembling techniques (“Snapshot”) with this reduced training time.
Since instability and nondeterminism are both a property of optimization, it is conceivable that use of a different optimizer might be able to lessen the degree of instability in model training.
We experimented SGD with various values of momentum, ranging from 0 for pure SGD to 0 .
999 for a momentum-driven optimizer, but none succeeded in reduce instability.
In addition, we experimented with the Adam (Kingma & Ba, 2014), as a representative of the class of adaptive learning rate algorithms, but this, too, had no effect on stability.
A GGRESSIVE S TOCHASTIC W EIGHT A VERAGING .
Inspired by the success found by Madhyastha & Jain (2019), we tried using Aggressive Stochastic Weight Averaging (ASWA), a variant of SWA (Izmailov et al., 2018).
However, we could not get the model to converge to a reasonable degree of [Table] performance with the original formulation due to update sizes that decreased too rapidly, and though we were able to modify it to converge successfully, the output variance remained as high as the other models
G RADIENT C LIPPING .
With the intuition that instability might be caused by spurious gradients of large magnitude, we experimented with clipping the norm of gradients (using pytorch ’s implementation of torch.nn.utils.clip_grad.clip_grad_norm_ .
Like other approaches, though, this had no effect on model variability.
W EIGHT A UGMENTATION
A very experimental approach, to reduce instability we experimented with taking an averaged gradient around the current set of parameters at each step, approximated by sampling a random weight offset before doing a forward or backward pass through the model.
Intuitively, this would encourage optimization to not be too sensitive to the current value of weights; however, in practice this simply didn’t affect the variance or stability of the model.
B A DDITIONAL L INEAR AND 2-L AYER R ESULTS
In Table 8 we include results for linear networks and 2-layer networks (1 hidden layer) on CIFAR-10, omitted from the main text due to space constraints.
Of note in these results is the relative smaller effect of a single bit change for a 2-layer network where the hidden layer is convolutional — still much larger than for the linear model, but signiﬁcantly smaller than for any other non-linear model. This suggests a similar effect to what was previously observed on MNIST (Table 6) in that degree of instability might be related to the interplay of model, dataset, and the degree of overﬁtting. C I MPACT OF R ANDOM B IT C HANGE O VER T IME In Fig.
3 we plot the effect of a random bit change for a linear and single on CIFAR-10, illustrating the effect of instability as described in Sec. 4. In the ﬁrst few epochs of training, we observe that the standard deviation and range of cross-entropy for the model with one hidden layer quickly grows, only eventually decreasing much later in training as the model’s parameters converges toward their ﬁnal values. On the other hand, for linear models, the standard deviation consistently remains 5 or more orders of magnitude lower throughout training. D S UBTLETIES IN E VALUATION
While we have done our best to present our experimental protocol as plainly as possible, one subtlety arises when interpreting results.
First, recall that in our example from Sec. 3.1, testing the effects [Figure] of random initialization corresponded to training models for ( S 1 , S 2 , S 3 ) ∈ { ( i, 1 , 1) } Ri =1 , where
S 1 is the seed for random initialization, S 2 is the seed for training data shufﬂing, and S 3 is set toto indicate the deterministic mode for cuDNN.
The subtlety arises in that the resulting distribution of ( S 1 , S 2 , S 3 ) ∈ { ( i, 1 , 1) } Ri =1 is not necessarily the same as the distribution where S 2 is set to a different arbitrary constant value, e.g. S 2 = 2 .
Due to this, there may be minor discrepancies when comparing the diversity in performance between two different sources of nondeterminism (albeit nothing likely to change general conclusions).
Combined with the natural sampling variability implicit in only training a ﬁnite number of models, this can lead to paradoxical results such as the standard deviation for a particular metric being slightly higher for a random bit change as compared to an entirely different random parameter initialization.
While we have separately validated that the general conclusions of our results hold when varying a few of these constant factors ( i.e. running experiments where S 2 is set to 2 and 3 , in this example), it is difﬁcult to resolve the discrepancy entirely without models according to the full cross-product of random seeds, which is prohibitive due to the exponential number of such models.