Reproducibility Report for "On Warm-Starting Neural Network Training"
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary Scope of Reproducibility
We reproduce the results of the paper "On Warm-Starting Neural Network Training.
" In many real-world applications,the training data is not readily available and is accumulated over time.
As training models from scratch is a timeconsuming task, it is preferred to use warm-starting i e., using the already existing models as the starting point to obtainfaster convergence.
This paper investigates the effect of warm-starting on the ﬁnal model’s performance.
It identiﬁes anoticeable gap between warm-started and randomly-initialized models, hereafter referenced as the warm-starting gap .
Furthermore, they provide a solution to mitigate this side-effect.
In addition to reproducing the original paper’s results,we propose an alternative solution and assess its effectiveness
Methodology
We reproduced almost every ﬁgure and table in the main text and some of those in the appendix.
We used ourimplementation to produce these results.
In case of a mismatch of the results, we also investigated the cause andproposed possible explanations.
We mainly used GPUs to train our models using infrastructure offered by public cloudsand those that were available to us privately.
Results
Most of our results closely match the reported results in the original paper.
Therefore, we conﬁrm that the warm-startinggap exists in certain settings and that the Shrink-Perturb method successfully reduces or eliminates this gap.
However,in some cases, we were not able to completely reproduce their results.
By investigating the root of such mismatches,we provide another solution to avoid this gap.
In particular, we show that data augmentation also helps to reduce thewarm-starting gap.
What was easy
The experiments described in the paper were based on regular training of neural networks on a portion of widely-useddatasets, possibly from a pre-trained model.
Therefore implementing each experiment was relatively easy to do.
Furthermore, since many of the parameters were reported in the original paper, we did not need much tuning in mostexperiments.
Finally, it is straightforward to implement and use the proposed solution
.What was difﬁcult
Though implementing each experiment is relatively simple, the numerosity of experiments proved to be slightlychallenging.
In particular, each of the online experiments in the original setting requires training a deep network toconvergence more than 30 times.
In these cases, we sometimes changed the settings, sacriﬁcing granularity to reducecomputation time.
However, these changes did not affect the interpretability of the ﬁnal results.
Communication with original authors
We brieﬂy communicated with the authors to clarify the experiments’ details, such as the convergence conditions.
Introduction
[Figure] Training large models from scratch is usually time and energy-consuming, so it is desired to have a method to accelerateretraining neural networks with new data added to the training set.
The well-known solution to this problem iswarm-starting.
Warm-Starting is the process of using the weights of a model, pre-trained on a subset of the data, as thestarting point of training with the complete data.
The paper investigates the effect of warm-starting on the ﬁnal model’s accuracy and identiﬁes a generalization gap onwarm-started models.
The paper also provides a method to mitigate this gap by shrinking the pre-trained weights andadding a random perturbation.
In this report, we repeat the original paper’s experiments and compare them with the reported results.
Also, we extendthe original paper results by investigating the effect of data augmentation on this phenomenon.
In particular, we establishthat using data augmentation might be a second solution to mitigating the generalization gap.
We report and discuss our results in Section 2.
In section 3, we detail our experimental settings and hyperparameters.
Results & Discussion
[Figure] 2.1 Warm-Starting Generalization Gap
[Figure]
Similar to the paper, we start by demonstrating the existence of a generalization gap when using warm-starting beforetraining.
We use the same set of experiments used by the authors.
Unless otherwise stated, we follow the settingsdescribed in the paper for our experiments.
In particular, in the ofﬂine setting, we ﬁrst train our model on half of the training data and then further train thepre-trained model on the whole dataset.
We compare the resulting model with a model trained from randomly initializedweights.
Figure 1 depicts the test accuracy of ResNet-18 [1] during training in this setting and matches Figure 1 of thepaper.
[Figure]
of training where the warm-started model is trained on the whole dataset.
We repeat this experiment with different datasets, models, andoptimizers.
In particular we perform experiments on CIFAR-10[2], CIFAR-100 [2], and SVHN [3].
As our model, we experiment with ResNet-18, a three-layer perceptron, and logisticregression.
The same models and datasets were used in theoriginal paper.
For the optimizers, we compare SGD [4] and
Adam [5].
In this particular experiment, we compare SGDwith and without momentum.
In the rest of this work, unlessexplicitly stated, SGD is used without momentum.
The ﬁnalaccuracies are reported in Table 1 similar to Table 1 of theoriginal paper.
Our results are similar to the paper’s results for
CIFAR-10 and CIFAR-100 datasets.
In particular, we observe ageneralization gap when using a warm-started model instead oftraining from scratch.
However, we did not observe the samegap on the SVHN dataset.
Furthermore, we were unable toobtain a reasonable accuracy with the MLP model using SGDwithout momentum with the reported setting on SVHN.
Weinstead report the result of using 0 .
005 learning rate.
We didnot perform hyperparameter tuning for the other experiments.
In the online setting, we follow the original paper and train ourmodel in several steps, increasing the amount of data availableat each step.
This setting is a more accurate simulation of the real-world problems where the training data grows overtime.
We split the training data into batches of 1000 samples and start adding them, one by one, to the pool of available data.
We follow two different scenarios.
In one scenario, we reinitialize our model randomly after each batch is added andtrain it from scratch.
In the other scenario, we continue training the model with the parameters learned in the previousstep.
After each batch is added, we continue training our model until convergence before adding the next batch.
We assumeconvergence when the model reaches 99% training accuracy.
By communicating with
As in the paper, we optimize the model using Adam optimizer with a learning rate of 0.001 on CIFAR-10.
Given thediscrepancy of our results on the SVHN datasets in the ofﬂine settings, we additionally perform the same experiment onthis dataset.
For the CIFAR-10 dataset, the generalization gap between random-initialization training and warm-starttraining is clearly observed (Figure 2a).
However, like the ofﬂine experiment, we did not observe the gap for the SVHN dataset (Figure 2b).
Still, we wereable to reproduce the gap by increasing the convergence accuracy threshold to 99 .
9% (Figure 2c).
Note that 99% trainaccuracy is more challenging to achieve on the CIFAR-10 dataset than on SVHN and therefore requires more training,possibly leading to more over-ﬁtting.
Increasing the convergence threshold compensates for this difference.
[Table]
This result and the fact that the authors show that the proposed
Shrink-Perturb method is similar to an aggressive regularization brings up the question of whether this gap might be a side-effect of over-ﬁtting when training on partial data.
There are various known techniques to prevent overﬁtting.
The original paper investigates the effect of some of these techniques, namelyregularization, and early-stopping.
We reproduced these experiments and explained the results below.
Early-Stopping : To investigate the effect of early-stopping,following the original paper, we trained a ResNet-18 model onhalf of the CIFAR-10 data and checkpointed its parameters every 20 epochs.
The result is plotted in Figure 3, which matches
Figure 4 of the original paper and shows the warm-starting gapcan be observed even after 20 epochs of training.
To decreasecomputational costs, we used lower granularity than the original paper to perform this experiment, saving parameters every20 epochs rather than 5 epochs.
Also, we only perform eachexperiment once.
Regularization : Regularization is commonly used to improvegeneralization.
The original paper explores the effect of varioustypes of regularization.
Due to time and resource limitations,we only look into weight decay, which is widely used andis a de-facto standard for training the state-of-the-art models.
We repeat the ofﬂine setting experiment on CIFAR-10 with aweight decay of 0 .
1 on both the pre-training and main training.
However, contrary to the original paper’s results, we observethat the warm-starting gap decreases when applying weightdecay.
We also test with weight decay values of 0 .
01 and 0 .
We ﬁnd out that higher values of weight decay result in lowerwarm-starting gap.
The results are reported in Table 2, whichcorresponds to Appendix Table 13 of the original paper.
Data Augmentation : Data augmentation is widely used toobtain state-of-the-art performance and is known to help generalization [6], but it is not used in the other experiments of thispaper.
It is speciﬁcally important to check the effect of dataaugmentation since it is widely used in practice.
Therefore weextend the original paper’s experiments by investigating the impact of data augmentation.
We report our results in Section 2.4.
2.2 Effect of Hyperparametes
Our results show that in some cases, a generalization gap existswhen pre-training our model on a portion of the ﬁnal dataset.
However, when training in the online setting on SVHN, wecould only observe this gap with a high enough convergence threshold.
The paper investigates the effect of othertraining hyperparameters, namely learning rate and batch size.
To investigate the effect of learning rate and batch size, we train a ResNet-18 with different values for these hyperparameters.
We choose the learning rate from { 0 . 1 , 0 . 01 , 0 . 001 } and the batch size from { 128 , 64 , 32 , 16 } .
We iterate
[Figure]
(a) For the CIFAR-10 dataset, the generalization gap be-
tween randomly initialized and warm started models is observed for the 99.0% convergence threshold.
However, the training time required at each step increases almost linearly in the case of randomly initialized model, but is constant when using warm-starting.
[Figure]
(b) For the SVHN dataset, there is no generalization gap between randomly initialized and warm
started models with the 99.0% convergence thresh- old.
[Figure]
(c) For SVHN, the generaliza tion gap appears by increasing the convergence threshold to 99.9%.
[Table]
over all pairs for these values.
For each pair, we train over the full CIFAR-10.
We also train a different model over50% of the CIFAR-10 dataset and use it to warm-start a model and train it on the whole dataset.
We use a differentlearning rate and batch size, randomly chosen from the sets of values, in the second part of the training, i.e., for trainingthe warm-started model.
We repeat each experiment 9 times.
Each model is trained to 99% training accuracy.
Thetest accuracy is plotted against training time in Figure 4, which corresponds with Figure 3 of the paper.
Note that thetraining time for the warm-started model corresponds to the time of the second part of the training.
In other words, thetime of training on half of the dataset is not included.
This is justiﬁed because the goal is to assess if warm-startingleads to comparable accuracy while saving training time when a new batch of data arrives.
In our results, choosing theright hyperparameters can lead to achieving comparable or even better accuracy, when using warm-starting, faster thantraining a randomly initialized model.
This does not match the results of the paper, where the warm-started models withcomparable accuracy take the same amount of training time as the randomly initialized models.
While we perform lessexperiments in the warm-started setting, we perform the same number of experiments with random initialization asdescribed in the original paper’s text.
However, the number of points in Figure 3 of the original paper corresponding torandomly initialized models, is more than what has been described in the text.
We note that performing more randomlyinitialized experiments might be the reason for the mismatch in our results with the original paper.
2.3 Shrink & Perturb Solution
In addition to establishing the warm-starting gap’s existence and investigating its roots, the paper also provides a methodto mitigate this issue.
In this method, the training starts from a shrunk and perturbed version of the pre-trained weights,so we reference it as the Shrink-Perturb method.
More speciﬁcally, for a given λ and σ , the new weight is computed as
[Equation]
where w old is the pre-trained weight and w random is the corresponding weight from a randomly initialized model.
Whenever we apply the Shrink-Perturb transformation, we create a new randomly initialized model and use its weightsas w random .
We tested the effectiveness of this method in both ofﬂine and online settings.
In the ofﬂine setting, we applied the
Shrink-Perturb transform after pre-training on 50% of CIFAR-10.
We used σ = 10 − 4 and repeated this experimentwith different values of λ .
We plotted the test accuracy during training on all of the data in Figure 5.
It can be seen thatthe method is effective and leads to even better performance than the randomly initialized model.
[Equation] [Figure] [Figure]
[Figure]
In the online setting
[Figure]
we applied the Shrink-Perturb transform every time a new batch of data is added.
To reducecomputation cost, we add data in batches of 2500 samples.
The result for σ = 10 − 4 and different values of λ is plottedin Figure 6.
Figure 6 matches Figure 7 of the original paper.
The result of applying the Shrink-Perturb method in theofﬂine setting is not reported in the original paper
[Figure]
[Figure]
To assess the impact of shrinking weights on the model’s performance,we ﬁt different models to CIFAR-10.
Then, we shrink the weightwith different values of λ and evaluate the accuracy.
Similar to thepaper, we train ResNet18 and an MLP with ReLU activation withand without bias.
In addition, we also train an MLP with Tanhactivation with and without bias.
The result is shown in Figure 7,which corresponds with Figure 6 of the paper.
The only differencein our ﬁndings with the original paper’s is that we observe classiﬁerperformance damage for MLP with ReLU for λ > 0 . 6 .
Thoughfor λ > 0 .
8 the damage is negligible.
Also, note that shrinking theweights of an MLP without bias and ReLU activation only scales theﬁnal output, which does not affect the output labels.
Therefore itsimmunity to shrinkage is expected.
The more interesting result is thateven for ResNet-18 or MLP with Tanh activation, the test accuracyis not signiﬁcantly damaged for λ values greater than 0.2.
In order to explain why the Shrink-Perturb method is effective, theoriginal paper compares the average gradients over the ﬁrst and sec-
ond half of the dataset during the training of the warm-started model in the ofﬂine setting.
In particular, a ResNet-18 isﬁrst trained on half of CIFAR-10.
Warm-starting from the pre-trained model, the model is trained on the full datasetwhile measuring the average gradient over the ﬁrst and second half of the dataset simultaneously.
It is observed that thegradient for the ﬁrst half, which the model was pre-trained on, is substantially lower than for the second part.
However,applying the Shrink-Perturb transformation eliminates this difference.
We reproduced this experiment with some slightmodiﬁcations.
In particular, instead of averaging the gradient over part of the dataset after each batch, we did it at thebeginning of each epoch.
We plotted these values in Figure 8.
Our result matches Figure 5 of the original paper.
Inparticular, we conﬁrm that the Shrink-Perturb method successfully eliminates the gap between the gradients.
2.4 Effect of Data Augmentation
Data augmentation is widely used in practice.
However, it is not used for the experiments in the original paper.
Therefore, we decided to assess its impact on the generalization gap for warm starting training.
We perform our experiments on ResNet-18 and CIFAR-10.
To augment the data, we ﬁrst pad the image with 4 pixels oneach side and then randomly crop it back to 32x32.
We then perform a random horizontal ﬂip with probability 0 .
Wealso apply color jitter with brightness, contrast, and saturation factor equal to 0 .
Finally, we also apply small randomrotations.
All experiments were done with SGD and a learning rate equal to 0.001 in order to make the setup consistentwith previous warm start experiments.
The results are reported on Figure 9 .
It can be seen that applying the augmentation mitigates the warm-starting gap.
We allow the models to train for 350epochs.
However, because the learning rate is low, the models are not fully converged even after 350 epochs.
We did notcontinue the training because of resource limitations.
However, it is visible that warm-starting with data augmentationcan achieve good performance faster than training from scratch.
To explain the effectiveness of the Shrink-Perturb solution, the original paper’s authors looked at the differences of thegradient norm for the ﬁrst and the second part of the dataset, which is heavily reduced after applying Shrink-Perturb (asshown in Figure 8).
Following the same direction, we checked if applying data augmentation can affect the differenceas well.
It can be seen in Figure 10 that, similar to the shrink perturb method, the gradient norm difference is alsomitigated when using data augmentation.
It is clear that applying data augmentation prevented overﬁtting.
The original warm start setup has a large divergencebetween train and test accuracy from the beginning of the training.
On the contrary, the model trained with dataaugmentation has close performance on train and test datasets.
We leave the investigation of other overﬁtting preventiontechniques’ effects as future work.
Additionally, we note that data augmentation usually slows down the convergence and it cannot be applied to every tasksince for some types of data transform set cannot be deﬁned.
Due to the limits of this report, we also leave the carefulcomparison between data augmentations and Shrink & Perturb as future research in this area.
2.5 Warm-Starting Gap in Transfer Learning
Deep learning models require large training sets to perform well.
This presents a problem in many practical caseswhere only limited data is available, and acquiring additional data is expensive.
This has encouraged the use of transferlearning [7; 8; 9]; the practice of warm-starting from a model trained on a different dataset.
[Figure]
To investigate whether a similar gap is observed in transfer learning
[Figure]
we trained a ResNet-18 model on one dataset andused the pre-trained weights to warm-start training on a different dataset.
We performed this experiment for all pairs of
CIFAR-100, CIFAR-10, and SVHN datasets.
To also investigate the effect of the amount of the data available, we alsoconsidered subsets of these datasets where only a fraction p of data is available.
More accurately, for every two datasetsand p ∈ { 0 . 1 , 0 . 3 , 0 . 6 , 1 . 0 } , we chose a random subset from each dataset containing p × n data points
[Figure]
where n is thetotal number of data points in that dataset.
We performed the described transfer learning experiment for these subsetsand recorded the ﬁnal test accuracy.
To assess the effect of warm-starting and the Shrink-Perturb method, we plottedthe ﬁnal test accuracy of each of the described three settings (random initialization, warm starting, and warm-startingwith Shrink-Perturb) with respect to p in Figure 11.
In this experiment, we used Adam as our optimizer.
This ﬁgurecorresponds with Figure 9 of the original paper.
At each part of the training, we train our models for 200 epochs.
When training CIFAR-100 starting from weights of amodel trained on SVHN or CIFAR-10, the last layer is initialized randomly because of the mismatch in the number ofclasses.
It can be seen that, as mentioned in the paper, the warm-starting gap exists in the transfer learning settings aswell, and that it is worsened when the amount of data available is increased.
Furthermore, the Shrink-Perturb methodproves useful in this setting, as well.
Methodology
In this section, we deﬁne the setting we used for our experiments.
There was no available code for the original paper,and we implemented everything from scratch.
We use the PyTorch framework for the implementations.
3.1 Model descriptions
Most of the experiments are performed using ResNet-18 [1].
Some experiments are also performed on a Multi-Layered
Perceptron (MLP) and Logistic Regression.
We detailed the structure of each of these models below.
• ResNet-18: We used an implementation of ResNet-18 tuned for CIFAR-10 dataset.
We used the code fromhttps://github.com/huyvnphan/PyTorch_CIFAR-10 .
In all experiments, batch normalization [10] wasenabled.
• MLP:
The MLP has three hidden layers, each of which has 100 neurons.
Either ReLU or Tanh was used asthe activation function.
Unless explicitly stated, the bias term is added.
• Logistic Regression: We implement Logistic Regression as a Multi-Layered Perceptron with no hiddenlayers.
We used either Adam or SGD optimizers for training the models.
More accurately, we use Adam in Table 1, Figure 2,
Figure 6, and Figure 11.
In all other experiments, we use SGD.
We use 0 .
001 learning rate and batches of size 128 .
Unless otherwise stated, we used SGD without momentum and without weight decay.
In cases where momentum wasused (such as in Table 1), the value of momentum was set to 0 .
The Adam optimizer was used with default parametersfrom PyTorch’s implementation, namely β 1 = 0 .
9 , and β 2 = 0 .
3.2 Datasets
Same as the original paper, we perform experiments on CIFAR-10, CIFAR-100, and SVHN datasets.
We normalizeeach of the RGB channels by the mean and standard deviation of that channel in the CIFAR-10 dataset.
Except for thedata augmentation experiments, we do not apply any data augmentation.
3.3 Hyperparameters
We used hyperparameters stated in the original paper in most of our experiments.
In cases where we deviated from thereported values, mostly due to computational resource and time limitation, we have reported them in the text where wedescribed the experiment.
In case a hyperparameter is not reported in the original paper, we either communicated withthe authors to ask the hyperparameters, pick a value making reasonable assumptions, or try out different values andreport the result for all of them.
In all these cases, we clariﬁed the parameter we used in the text.
3.4 Experimental setup
We ran our experiments on both public cloud infrastructure, such as Google Colab and private GPUs that wereavailable to us.
Therefore the infrastructure varies between different experiments.
Our implementations for all theexperiments in this work is available in the Supplementary Material and also in https://github.com/CS-433/cs-433-project-2-fesenjoon .
Communication with Authors
In the original paper [11], it was not clear what convergence condition was used to stop the training.
Therefore, Wecommunicated with the authors via email and asked them to explain the convergence condition used in the experimentsmore clearly.
They stated that convergence happens when the training accuracy reaches 99%.
However, for reproducingthe Table 1 which also uses simpler models like Logistic Regression and MLP, it is not possible to reach 99% accuracy.
They clariﬁed that in this scenario, the convergence condition is met when the training accuracy stops improving.
Conclusion
We have veriﬁed the existence of the generalization gap in certain training settings.
Additionally, we have conﬁrmedthat the introduced Shrink-Perturb method can be effective in removing this gap.
We did this by repeating experimentsof the original paper and performing some experiments of our own.
However, we also encountered cases where wewere not able to reproduce the warm-starting gap or where the Shrink-Perturb method was not very successful.
Inaddition, we reproduced several experiments to investigate the effect of hyper-parameters, such as learning rate, onthis phenomenon.
Finally, we have shown that applying data augmentation can also help to remove this gap.
To allowothers to reproduce our results, we have detailed our experiments and have released our code.