Original: O N THE D IFFICULTY OF W ARM -S TARTING N EURAL N ETWORK T RAINING
Revised: O N THE D IFFICULTY OF W ARM -S TARTING N EURAL N ETWORK T RAINING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In many real-world deployments of machine learning systems, data arrive piecemeal.
Revised: In many real-world deployments of machine learning systems, data arrive piecemeal.

Original: These learning scenarios may be passive, where data arrive incrementally due to structural properties of the problem (e.g., daily ﬁnancial data) or active, where samples are selected according to a measure of their quality (e.g., experimental design).
Revised: These learning scenarios may be passive, where data arrive incrementally due to structural properties of the problem (e.g., daily ﬁnancial data) or active, where samples are selected according to a measure of their quality (e.g., experimental design).

Original: In both of these cases, we are building a sequence of models that incorporate an increasing amount of data.
Revised: In both of these cases, we are building a sequence of models that incorporate an increasing amount of data.

Original: We would like each of these models in the sequence to be performant and take advantage of all the data that are available to that point.
Revised: We would like each of these models in the sequence to be performant and take advantage of all the data that are available to that point.

Original: Conventional intuition suggests that when solving a sequence of related optimization problems of this form, it should be possible to initialize using the solution of the previous iterate—to “warm start” the optimization rather than initialize from scratch—and see reductions in wall-clock time.
Revised: Conventional intuition suggests that when solving a sequence of related optimization problems of this form, it should be possible to initialize using the solution of the previous iterate—to “warm start” the optimization rather than initialize from scratch—and see reductions in wall-clock time.

Original: However, in practice this warm-starting seems to yield poorer generalization performance than models that have fresh random initializations, even though the ﬁnal training losses are similar.
Revised: However, in practice this warm-starting seems to yield poorer generalization performance than models that have fresh random initializations, even though the ﬁnal training losses are similar.

Original: While it appears that some hyperparameter settings allow a practitioner to close this generalization gap, they seem to only do so in regimes that damage the wall-clock gains of the warm start.
Revised: While it appears that some hyperparameter settings allow a practitioner to close this generalization gap, they seem to only do so in regimes that damage the wall-clock gains of the warm start.

Original: Nevertheless, it is highly desirable to be able to warm-start neural network training, as it would dramatically reduce the resource usage associated with the construction of performant deep learning systems.
Revised: Nevertheless, it is highly desirable to be able to warm-start neural network training, as it would dramatically reduce the resource usage associated with the construction of performant deep learning systems.

Original: In this work, we take a closer look at this empirical phenomenon and try to understand when and how it occurs.
Revised: In this work, we take a closer look at this empirical phenomenon and try to understand when and how it occurs.

Original: Although the present investigation did not lead to a solution, we hope that a thorough articulation of the problem will spur new research that may lead to improved methods that consume fewer resources during training.
Revised: Although the present investigation did not lead to a solution, we hope that a thorough articulation of the problem will spur new research that may lead to improved methods that consume fewer resources during training.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Although machine learning research generally assumes the existence of a ﬁxed set of training data, real-life machine learning systems face more complicated situations.
Revised: Although machine learning research generally assumes the existence of a ﬁxed set of training data, real-life machine learning systems face more complicated situations.

Original: One particularly common scenario is where a production machine learning system must be constantly updated with new data.
Revised: One particularly common scenario is where a production machine learning system must be constantly updated with new data.

Original: This situation occurs in ﬁnance, online advertising, recommendation systems, fraud detection, and many other domains where machine learning systems are used for prediction and decision making in the real world (He et al., 2014;
Revised: This situation occurs in ﬁnance, online advertising, recommendation systems, fraud detection, and many other domains where machine learning systems are used for prediction and decision making in the real world (He et al., 2014;

Original: Chandramouli et al., 2011; Kuncheva, 2008).
Revised: Chandramouli et al., 2011; Kuncheva, 2008).

Original: When the new data arrive, the model needs to be updated so that it can be as accurate as possible and to also account for any domain shift that is occurring.
Revised: When the new data arrive, the model needs to be updated so that it can be as accurate as possible and to also account for any domain shift that is occurring.

Original: As a concrete example, consider a large-scale social media website, to which users are constantly uploading images and text.
Revised: As a concrete example, consider a large-scale social media website, to which users are constantly uploading images and text.

Original: The company would like to have up-to-the-minute predictive models in order to recommend content, ﬁlter out inappropriate media, and determine what advertisements to show.
Revised: The company would like to have up-to-the-minute predictive models in order to recommend content, ﬁlter out inappropriate media, and determine what advertisements to show.

Original: There might be millions of new data arriving every day, the structure of which needs to be rapidly incorporated into the production machine learning systems.
Revised: There might be millions of new data arriving every day, the structure of which needs to be rapidly incorporated into the production machine learning systems.

Original: It is natural in this scenario to imagine maintaining, in effect, a single model that is updated with the latest data at a regular cadence.
Revised: It is natural in this scenario to imagine maintaining, in effect, a single model that is updated with the latest data at a regular cadence.

Original: Every day, for example, new training might be performed on the model with the updated, larger data set.
Revised: Every day, for example, new training might be performed on the model with the updated, larger data set.

Original: Ideally, this new training procedure is initialized from the parameters of yesterday’s model, i.e., it is “warm-started” from those parameters rather than a fresh initialization.
Revised: Ideally, this new training procedure is initialized from the parameters of yesterday’s model, i.e., it is “warm-started” from those parameters rather than a fresh initialization.

Original: Such an initialization makes intuitive sense: the data used yesterday are mostly the same as the data today, and it seems wasteful to throw away all the previous computation.
Revised: Such an initialization makes intuitive sense: the data used yesterday are mostly the same as the data today, and it seems wasteful to throw away all the previous computation.

Original: For linear models and convex optimization problems, such warm starting is widely used and highly successful, e.g., He et al.
Revised: For linear models and convex optimization problems, such warm starting is widely used and highly successful, e.g., He et al.

Original: ; the theoretical properties of online learning are well understood for such problems.
Revised: ; the theoretical properties of online learning are well understood for such problems.

Original: However, warm-starting seems to hurt generalization in deep neural networks.
Revised: However, warm-starting seems to hurt generalization in deep neural networks.

Original: This is particularly troubling, because warm-starting does not damage training accuracy.
Revised: This is particularly troubling, because warm-starting does not damage training accuracy.

Original: Figure 1 shows a simple illustration of this phenomenon.
Revised: Figure 1 shows a simple illustration of this phenomenon.

Original: Three 18-layer ResNets have been trained on the
Revised: Three 18-layer ResNets have been trained on the

Original: CIFAR-10 natural image classiﬁcation task to create these ﬁgures.
Revised: CIFAR-10 natural image classiﬁcation task to create these ﬁgures.

Original: One was trained on 100% of the data, one was trained on 50% of the data, and a third warm-started model was trained on 100% of the data but initialized from the parameters found from the 50% training.
Revised: One was trained on 100% of the data, one was trained on 50% of the data, and a third warm-started model was trained on 100% of the data but initialized from the parameters found from the 50% training.

Original: All three achieve the upper bound on training accuracy.
Revised: All three achieve the upper bound on training accuracy.

Original: However, the warm-started network performs worse on test samples than the network trained on the same data but with a good random initialization.
Revised: However, the warm-started network performs worse on test samples than the network trained on the same data but with a good random initialization.

Original: Problematically, this phenomenon incentivizes performancefocused researchers and engineers to constantly retrain models from scratch , at potentially enormous ﬁnancial and environmental cost (Strubell et al., 2019); this is an example of “Red AI” as articulated in
Revised: Problematically, this phenomenon incentivizes performancefocused researchers and engineers to constantly retrain models from scratch , at potentially enormous ﬁnancial and environmental cost (Strubell et al., 2019); this is an example of “Red AI” as articulated in

Original: Schwartz et al.
Revised: Schwartz et al.

Original: (2019) , which disregards resource consumption in the pursuit of raw predictive performance.
Revised: (2019) , which disregards resource consumption in the pursuit of raw predictive performance.

Original: We observe that the warm-starting phenomenon has implications for other situations of interest as well.
Revised: We observe that the warm-starting phenomenon has implications for other situations of interest as well.

Original: In active learning, for example, unlabeled samples are abundant but the labels themselves are expensive.
Revised: In active learning, for example, unlabeled samples are abundant but the labels themselves are expensive.

Original: The goal of the learner is to identify samples that would maximally improve its hypothesis, then send those to an oracle to be labeled and integrated into the existing training set.
Revised: The goal of the learner is to identify samples that would maximally improve its hypothesis, then send those to an oracle to be labeled and integrated into the existing training set.

Original: These decisions are best made using a model that has been ﬁtted to every item in the training set; a model that is trained on samples that were chosen one at a time will almost always outperform a model that is trained on samples that were chosen in batches (Yang and Carbonell, 2013).
Revised: These decisions are best made using a model that has been ﬁtted to every item in the training set; a model that is trained on samples that were chosen one at a time will almost always outperform a model that is trained on samples that were chosen in batches (Yang and Carbonell, 2013).

Original: It would be time efﬁcient, then, to simply update the model each time a sample is appended to the training set, rather than retrain the model from scratch at every iteration of sample acquisition.
Revised: It would be time efﬁcient, then, to simply update the model each time a sample is appended to the training set, rather than retrain the model from scratch at every iteration of sample acquisition.

Original: However, this kind of update seems to damage generalization.
Revised: However, this kind of update seems to damage generalization.

Original: In fact, although this phenomenon has not received much direct attention from the research community, some recent papers on deep active learning mention that their model is retrained at every step of data collection (Sener and Savarese, 2018; Ash et al., 2019), and popular deep active learning repositories on Github (Rostamiz, 2017–2019; Huang, 2018–2019) all retrain models from scratch.
Revised: In fact, although this phenomenon has not received much direct attention from the research community, some recent papers on deep active learning mention that their model is retrained at every step of data collection (Sener and Savarese, 2018; Ash et al., 2019), and popular deep active learning repositories on Github (Rostamiz, 2017–2019; Huang, 2018–2019) all retrain models from scratch.

Original: This caveat limits the feasibility of active learning for deep neural networks in the absence of the ability to effectively warm-start training.
Revised: This caveat limits the feasibility of active learning for deep neural networks in the absence of the ability to effectively warm-start training.

Original: The ineffectiveness of warm-starting has been observed anecdotally in the community, but this paper seeks to examine its empirical properties more closely in controlled settings.
Revised: The ineffectiveness of warm-starting has been observed anecdotally in the community, but this paper seeks to examine its empirical properties more closely in controlled settings.

Original: Note that the ﬁndings in this paper are not inconsistent with extensive work on unsupervised pre-training (Erhan et al., 2010; Bengio, 2011) 
Revised: Note that the ﬁndings in this paper are not inconsistent with extensive work on unsupervised pre-training (Erhan et al., 2010; Bengio, 2011) and transfer learning in the small-data and “few shot” problem regimes (Vinyals et al., 2016; Snell et al.,

Original: [Table]
Revised: [Table]

Original: 9 (0.3) and transfer learning in the small-data and “few shot” problem regimes (Vinyals et al., 2016; Snell et al., 2017; Finn et al., 2017).
Revised: ; Finn et al., 2017).

Original: Rather here we are examining how to accelerate training in the large-data supervised regime in a way that is consistent with expectations from convex problems.
Revised: Rather here we are examining how to accelerate training in the large-data supervised regime in a way that is consistent with expectations from convex problems.

Original: The ultimate goal is to achieve an understanding that can lead to procedures for successful warm starting and reduced resource usage in common real-world retraining scenarios.
Revised: The ultimate goal is to achieve an understanding that can lead to procedures for successful warm starting and reduced resource usage in common real-world retraining scenarios.

Original: This paper is fundamentally about investigating and reporting on a phenomenon that should be of broad interest to researchers and practitioners.
Revised: This paper is fundamentally about investigating and reporting on a phenomenon that should be of broad interest to researchers and practitioners.

Original: Our investigation has not yet revealed a solution, but we hope it will spur a broader discussion about techniques for reducing the resource consumption of deep neural network training.
Revised: Our investigation has not yet revealed a solution, but we hope it will spur a broader discussion about techniques for reducing the resource consumption of deep neural network training.

Original:  W ARM S TARTING D AMAGES G ENERALIZATION
Revised:  W ARM S TARTING D AMAGES G ENERALIZATION

Original: In this section we provide empirical evidence that warm starting consistently damages generalization performance in neural networks.
Revised: In this section we provide empirical evidence that warm starting consistently damages generalization performance in neural networks.

Original: To that end, we conduct a series of experiments across a several different architectures, optimizers, and image datasets.
Revised: To that end, we conduct a series of experiments across a several different architectures, optimizers, and image datasets.

Original: Our goal is to create simple, reproducible settings that are reﬂective of real-world situations, in which the warm-starting phenomenon is observable.
Revised: Our goal is to create simple, reproducible settings that are reﬂective of real-world situations, in which the warm-starting phenomenon is observable.

Original: 2. B ASIC B ATCH U PDATING
Revised: 2. B ASIC B ATCH U PDATING

Original: In order to clearly highlight the basic problem, we consider the simplest case of warm-starting, in which a single training dataset is partitioned into two subsets that are presented sequentially.
Revised: In order to clearly highlight the basic problem, we consider the simplest case of warm-starting, in which a single training dataset is partitioned into two subsets that are presented sequentially.

Original: In each series of experiments, we randomly segment the training data into two batches.
Revised: In each series of experiments, we randomly segment the training data into two batches.

Original: The model is trained to convergence on the ﬁrst half, then is trained on the union of the two batches, i.e., 100% of the data.
Revised: The model is trained to convergence on the ﬁrst half, then is trained on the union of the two batches, i.e., 100% of the data.

Original: This is repeated for modern deep learning classiﬁers (ResNet-18 of He et al.
Revised: This is repeated for modern deep learning classiﬁers (ResNet-18 of He et al.

Original: (2016), multilayer perceptrons (MLPs) consisting of three layers and tanh activations, and logistic regression.
Revised: (2016), multilayer perceptrons (MLPs) consisting of three layers and tanh activations, and logistic regression.

Original: Models are optimized using either stochastic gradient descent or the Adam variant of stochastic gradient descent (Kingma and Ba, 2014), and are ﬁtted to the standard CIFAR-10, CIFAR-100, and SVHN classiﬁcation datasets.
Revised: Models are optimized using either stochastic gradient descent or the Adam variant of stochastic gradient descent (Kingma and Ba, 2014), and are ﬁtted to the standard CIFAR-10, CIFAR-100, and SVHN classiﬁcation datasets.

Original: All models are trained using a mini-batch size of 128 and a learning rate of 0.001, the smallest learning rate used in the learning schedule for ﬁtting stateof-the-art ResNet models (He et al., 2016).
Revised: All models are trained using a mini-batch size of 128 and a learning rate of 0.001, the smallest learning rate used in the learning schedule for ﬁtting stateof-the-art ResNet models (He et al., 2016).

Original: We further investigate the effect of these parameters in Section 3.
Revised: We further investigate the effect of these parameters in Section 3.

Original: Our results (shown in Table 1) indicate that generalization performance is damaged consistently and signiﬁcantly for both ResNets and MLPs.
Revised: Our results (shown in Table 1) indicate that generalization performance is damaged consistently and signiﬁcantly for both ResNets and MLPs.

Original: This effect is more dramatic when data are more challenging to model, as in CIFAR-10, for which obtaining competitive validation performance generally requires data augmentation and a sophisticated architecture.
Revised: This effect is more dramatic when data are more challenging to model, as in CIFAR-10, for which obtaining competitive validation performance generally requires data augmentation and a sophisticated architecture.

Original: When the classiﬁcation problem is believed to be relatively easy, as in SVHN, the warm-start generalization gap is less pronounced.
Revised: When the classiﬁcation problem is believed to be relatively easy, as in SVHN, the warm-start generalization gap is less pronounced.

Original: Conversely, logistic regression, which enjoys a convex loss surface, is not signiﬁcantly damaged by warm starting.
Revised: Conversely, logistic regression, which enjoys a convex loss surface, is not signiﬁcantly damaged by warm starting.

Revised: Note that magnitude of the generalization gap is inversely proportional to the fraction of data available in the ﬁrst-round of training (Appendix Figure 8).

Original: This result is surprising.
Revised: This result is surprising.

Original: Even though MLP and ResNet optimization is non-convex, conventional intuition suggests that the warm-started solution should be close to the full-data solution and therefore a good initialization.
Revised: Even though MLP and ResNet optimization is non-convex, conventional intuition suggests that the warm-started solution should be close to the full-data solution and therefore a good initialization.

Original: One view on pre-training is that the initialization is a “prior” on weights; we often view prior distributions as arising from inference on old (or hypothetical) data and so this sort of pre-training should always be helpful.
Revised: One view on pre-training is that the initialization is a “prior” on weights; we often view prior distributions as arising from inference on old (or hypothetical) data and so this sort of pre-training should always be helpful.

Original: The generalization gap shown here creates a computational burden for real-life machine learning systems that must be retrained from scratch to perform well, rather than initialized from previous models.
Revised: The generalization gap shown here creates a computational burden for real-life machine learning systems that must be retrained from scratch to perform well, rather than initialized from previous models.

Original: 2. O NLINE L EARNING
Revised: 2. O NLINE L EARNING

Original: Passive Online Learning.
Revised: Passive Online Learning.

Original: A common real-world setting involves data that are being provided to the machine learning system in a stream.
Revised: A common real-world setting involves data that are being provided to the machine learning system in a stream.

Original: At every step, the learner is given k new samples to append to its training data, and it updates its hypothesis to reﬂect the larger dataset.
Revised: At every step, the learner is given k new samples to append to its training data, and it updates its hypothesis to reﬂect the larger dataset.

Original: Financial data, social media, and recommendations systems are common examples of scenarios where new data are constantly arriving.
Revised: Financial data, social media, and recommendations systems are common examples of scenarios where new data are constantly arriving.

Original: This paradigm is simulated in Figure 2, where we supply CIFAR-10 data, selected randomly without replacement, in groups of 100 to an 18-layer ResNet.
Revised: This paradigm is simulated in Figure 2, where we supply CIFAR-10 data, selected randomly without replacement, in groups of 100 to an 18-layer ResNet.

Original: We examine two cases: 1) where the model is retrained from scratch after each batch, starting from a random initialization, and 2) where the model is trained to convergence starting from the parameters learned in the previous iteration.
Revised: We examine two cases: 1) where the model is retrained from scratch after each batch, starting from a random initialization, and 2) where the model is trained to convergence starting from the parameters learned in the previous iteration.

Original: In both cases, the models are optimized with Adam, using an initial learning rate of 0.001.
Revised: In both cases, the models are optimized with Adam, using an initial learning rate of 0.001.

Original: Each was run ﬁve times with different random seeds and validation sets composed of a random third of the training data, reinitializing Adam’s parameters at each step of learning (see Section 3.6) .
Revised: Each was run ﬁve times with different random seeds and validation sets composed of a random third of the training data, reinitializing Adam’s parameters at each step of learning (see Section 3.6) .

Original: Figure 2 shows the trade-off between the two approaches.
Revised: Figure 2 shows the trade-off between the two approaches.

Original: The mean and standard deviations across the ﬁve runs are shown.
Revised: The mean and standard deviations across the ﬁve runs are shown.

Original: On the right are the training times: clearly, starting from the previous model is preferable and has the potential to vastly reduce computational costs and wall-clock time.
Revised: On the right are the training times: clearly, starting from the previous model is preferable and has the potential to vastly reduce computational costs and wall-clock time.

Original: However, as can be seen on the left, generalization performance is worse in the warm-started situation.
Revised: However, as can be seen on the left, generalization performance is worse in the warm-started situation.

Original: As more data arrive, the gap in validation accuracy increases substantially.
Revised: As more data arrive, the gap in validation accuracy increases substantially.

Original: Active Online Learning.
Revised: Active Online Learning.

Original: Instead of absorbing a stream of data, it may be possible to perform active learning (Mitchell et al., 1990) and choose what data to add to the dataset.
Revised: Instead of absorbing a stream of data, it may be possible to perform active learning (Mitchell et al., 1990) and choose what data to add to the dataset.

Original: When the data are expensive to acquire, it’s sometimes advantageous to select samples carefully.
Revised: When the data are expensive to acquire, it’s sometimes advantageous to select samples carefully.

Original: In this situation, the data are often chosen according to a metric that prefers data which are highly informative about the predictive hypothesis or which are likely to maximally reduce expected risk.
Revised: In this situation, the data are often chosen according to a metric that prefers data which are highly informative about the predictive hypothesis or which are likely to maximally reduce expected risk.

Original: Active learning is closely related to optimal experimental design in statistics.
Revised: Active learning is closely related to optimal experimental design in statistics.

Original: Generally, the setup is to allow the model to choose k new data samples for which it will be provided labels.
Revised: Generally, the setup is to allow the model to choose k new data samples for which it will be provided labels.

Original: These newly labeled data are then added to the training set and the model is updated.
Revised: These newly labeled data are then added to the training set and the model is updated.

Original: As in the passive scenario described in Section 2.2, we must choose whether to retrain the model from scratch with the new data or warm-start from the previous iteration.
Revised: As in the passive scenario described in Section 2.2, we must choose whether to retrain the model from scratch with the new data or warm-start from the previous iteration.

Original: While there are many algorithms for active learning, we study this scenario using margin-based sampling (Wang and Shang, 2014).
Revised: While there are many algorithms for active learning, we study this scenario using margin-based sampling (Wang and Shang, 2014).

Original: This approach quantiﬁes predictive uncertainty as inversely proportional to the difference between the model’s largest and second largest prediction.
Revised: This approach quantiﬁes predictive uncertainty as inversely proportional to the difference between the model’s largest and second largest prediction.

Original: As in Section 2.2, the algorithm receives 100 new training data at each round of learning, selected according to the margin criterion.
Revised: As in Section 2.2, the algorithm receives 100 new training data at each round of learning, selected according to the margin criterion.

Original: [Figure]
Revised: [Figure]

Original: Domain Shift.
Revised: Domain Shift.

Original: Figure 9 in the Appendix shows that active selection of data does not signiﬁcantly change the generalization gap between fresh initializations and warm starting.
Revised: Figure 10 in the Appendix shows that active selection of data does not signiﬁcantly change the generalization gap between fresh initializations and warm starting.

Original: Relative to the passive scenario, more data are required before the warm-started model starts to underperform, but a gap nevertheless appears.
Revised: Relative to the passive scenario, more data are required before the warm-started model starts to underperform, but a gap nevertheless appears.

Original: As before, this is over ﬁve random restarts and the right-hand subﬁgure shows the dramatic difference in computational cost.
Revised: As before, this is over ﬁve random restarts and the right-hand subﬁgure shows the dramatic difference in computational cost.

Original: Many online learning scenarios involve domain shift , in which the distribution is shifting as data stream in.
Revised: Many online learning scenarios involve domain shift , in which the distribution is shifting as data stream in.

Original: Such a shift could arise, for example, due to macroecomic changes effecting ﬁnancial markets, or seasonal changes inﬂuencing people’s tastes in a recommendation system.
Revised: Such a shift could arise, for example, due to macroecomic changes effecting ﬁnancial markets, or seasonal changes inﬂuencing people’s tastes in a recommendation system.

Original: In general, one would expect domain shift to make the warm-starting phenomenon worse, as the initialization is now even farther from the ﬁnal solution and indeed what the experiments show.
Revised: In general, one would expect domain shift to make the warm-starting phenomenon worse, as the initialization is now even farther from the ﬁnal solution and indeed what the experiments show.

Original: We simulate domain shift by sorting the data according to their ﬁrst principal component and supplying them to the learner in order.
Revised: We simulate domain shift by sorting the data according to their ﬁrst principal component and supplying them to the learner in order.

Original: This induces nonstationarity across batches, reﬂecting large-scale shifts in, e.g., illumination.
Revised: This induces nonstationarity across batches, reﬂecting large-scale shifts in, e.g., illumination.

Original: Figure 8 in the Appendix shows these deleterious effects.
Revised: Figure 9 in the Appendix shows these deleterious effects.

Original: 2. T RANSFER L EARNING
Revised: 2. T RANSFER L EARNING

Original: Despite successes on a variety of machine learning tasks, deep neural networks are still data hungry and generally require large training sets to generalize well.
Revised: Despite successes on a variety of machine learning tasks, deep neural networks are still data hungry and generally require large training sets to generalize well.

Original: For problems where only limited data are available, it has become popular to warm-start learning using the parameters from training on a different but related problem (Finn et al., 2017; Nichol and Schulman, 2018).
Revised: For problems where only limited data are available, it has become popular to warm-start learning using the parameters from training on a different but related problem (Finn et al., 2017; Nichol and Schulman, 2018).

Original: Transfer and “few-shot” learning in this form has seen success in computer vision and NLP (Mou et al., 2016).
Revised: Transfer and “few-shot” learning in this form has seen success in computer vision and NLP (Mou et al., 2016).

Original: The experiments we perform here, however, imply that when the second problem is not data-limited we would expect to see this transfer learning approach deteriorate in quality.
Revised: The experiments we perform here, however, imply that when the second problem is not data-limited we would expect to see this transfer learning approach deteriorate in quality.

Original: That is, at some point, the pre-training transfer learning approach is essentially the same as warm-starting under domain shift, and the generalization performance should suffer.
Revised: That is, at some point, the pre-training transfer learning approach is essentially the same as warm-starting under domain shift, and the generalization performance should suffer.

Original: We demonstrate this phenomenon by ﬁrst training a ResNet-18 to convergence on CIFAR-10, then using that solution to warm start a model trained on a varying fraction of the SVHN dataset.
Revised: We demonstrate this phenomenon by ﬁrst training a ResNet-18 to convergence on CIFAR-10, then using that solution to warm start a model trained on a varying fraction of the SVHN dataset.

Original: When only a small portion of SVHN is used, this is essentially the same as the pre-training transfer learning approach.
Revised: When only a small portion of SVHN is used, this is essentially the same as the pre-training transfer learning approach.

Original: As the proportion increases, the problem turns into what we have described here as warm-starting.
Revised: As the proportion increases, the problem turns into what we have described here as warm-starting.

Original: Figure 3 shows the result of this experiment, and it appears to support our intuition.
Revised: Figure 3 shows the result of this experiment, and it appears to support our intuition.

Original: When the second dataset is small, warm-starting is helpful, but there is a crossover point where better generalization would be achieved from training from scratch on that fraction of SVHN.
Revised: When the second dataset is small, warm-starting is helpful, but there is a crossover point where better generalization would be achieved from training from scratch on that fraction of SVHN.

Original: All models are optimized using SGD with a learning rate of 0.001.
Revised: All models are optimized using SGD with a learning rate of 0.001.

Original:  O VERCOMING THE W ARM -S TART P ROBLEM
Revised:  O VERCOMING THE W ARM -S TART P ROBLEM

Original: The design space for initializing and training deep neural network models is very large, and so it is important to evaluate whether there is some trick that could be used to help warm-started training ﬁnd good solutions.
Revised: The design space for initializing and training deep neural network models is very large, and so it is important to evaluate whether there is some trick that could be used to help warm-started training ﬁnd good solutions.

Original: Put another way, a reasonable response to this problem is “Did you see whether X helped?”
Revised: Put another way, a reasonable response to this problem is “Did you see whether X helped?”

Original: where X might be anything from batch normalization (Ioffe and Szegedy, 2015) to increasing the mini-batch size (Keskar et al., 2016).
Revised: where X might be anything from batch normalization (Ioffe and Szegedy, 2015) to increasing the mini-batch size (Keskar et al., 2016).

Original: This section tries to answer some of these questions and further empirically probe the warm-start phenomenon.
Revised: This section tries to answer some of these questions and further empirically probe the warm-start phenomenon.

Original: Unless otherwise stated, experiments in this section use a ResNet-18 model trained using SGD with a learning rate of 0.001 on CIFAR-10 data.
Revised: Unless otherwise stated, experiments in this section use a ResNet-18 model trained using SGD with a learning rate of 0.001 on CIFAR-10 data.

Original: All experiments were run ﬁve times to report means and variances.
Revised: All experiments were run ﬁve times to report means and variances.

Original: [Figure]
Revised: [Figure]

Original: 3. I S THIS AN EFFECT OF BATCH SIZE OR LEARNING RATE ?
Revised: 3. I S THIS AN EFFECT OF BATCH SIZE OR LEARNING RATE ?

Original: One might reasonably ask whether or not there exist any hyperparameters that close the generalization gap between warm started and randomly initialized models.
Revised: One might reasonably ask whether or not there exist any hyperparameters that close the generalization gap between warm started and randomly initialized models.

Original: In particular, can setting a larger learning rate at either the ﬁrst or second round of learning help the model escape to regions that generalize better?
Revised: In particular, can setting a larger learning rate at either the ﬁrst or second round of learning help the model escape to regions that generalize better?

Original: Can shrinking the batch size inject stochasticity that might improve generalization (Li et al., 2017; Gunasekar et al., 2017)?
Revised: Can shrinking the batch size inject stochasticity that might improve generalization (Li et al., 2018b; Gunasekar et al., 2017)?

Original: Here we again consider a warm-started experiment of training on 50% of CIFAR-10 until convergence, then training on 100% of CIFAR-10, using the initial round of training as an initialization.
Revised: Here we again consider a warm-started experiment of training on 50% of CIFAR-10 until convergence, then training on 100% of CIFAR-10, using the initial round of training as an initialization.

Original: We explore all combinations of batch sizes { 16 , 32 , 64 , 128 } , and learning rates { 0 . 001 , 0 . 01 , 0 . 1 } , varying them across the three rounds of training.
Revised: We explore all combinations of batch sizes { 16 , 32 , 64 , 128 } , and learning rates { 0 . 001 , 0 . 01 , 0 . 1 } , varying them across the three rounds of training.

Original: This allows for the possibility that there exist different hyperparameters for the ﬁrst stage of training that are better when used with a different set after warm-starting.
Revised: This allows for the possibility that there exist different hyperparameters for the ﬁrst stage of training that are better when used with a different set after warm-starting.

Original: Each of these combinations is run with three random initializations.
Revised: Each of these combinations is run with three random initializations.

Original: Figure 4 visualizes these results.
Revised: Figure 4 visualizes these results.

Original: Every resulting 100% model is shown from all three initializations and all combinations, with color indicating whether it was a random initialization or a warm-start.
Revised: Every resulting 100% model is shown from all three initializations and all combinations, with color indicating whether it was a random initialization or a warm-start.

Original: The horizontal axis shows the time to completion, excluding the pre-training time, and the vertical axis shows the resulting validation performance.
Revised: The horizontal axis shows the time to completion, excluding the pre-training time, and the vertical axis shows the resulting validation performance.

Original: Interestingly, we do ﬁnd warm-started models that perform as well as randomly-initialized models, but they are unable to do so while beneﬁting from their warm-started initialization.
Revised: Interestingly, we do ﬁnd warm-started models that perform as well as randomly-initialized models, but they are unable to do so while beneﬁting from their warm-started initialization.

Original: The training time for warm-started
Revised: The training time for warm-started

Original: ResNet models that generalize as well as randomly-initialized models is roughly the same as those randomlyinitialized models.
Revised: ResNet models that generalize as well as randomly-initialized models is roughly the same as those randomlyinitialized models.

Original: That is, there is no computational beneﬁt to using these warm-started initializations.
Revised: That is, there is no computational beneﬁt to using these warm-started initializations.

Original: It is worth noting that this plot does not capture the time or energy required to identify hyperparameters that close the generalization gap; such hyperparameter searches are often the culprit in the resource footprint of deep learning (Schwartz et al., 2019).
Revised: It is worth noting that this plot does not capture the time or energy required to identify hyperparameters that close the generalization gap; such hyperparameter searches are often the culprit in the resource footprint of deep learning (Schwartz et al., 2019).

Original: Wall-clock time is measured by assigning every model identical resources, which includes 50GB of RAM and an NVIDA Tesla P100 GPU.
Revised: Wall-clock time is measured by assigning every model identical resources, which includes 50GB of RAM and an NVIDIA Tesla P100 GPU.

Original: This increased ﬁtting time occurs because warm-started models, when using hyperparameters that generalize relatively well, seem to “forget” what was learned in the ﬁrst round of training.
Revised: This increased ﬁtting time occurs because warm-started models, when using hyperparameters that generalize relatively well, seem to “forget” what was learned in the ﬁrst round of training.

Original: Figure 5 demonstrates this phenomenon by computing the Pearson correlation between the weights of converged warm-started models and their initialization weights, again across various choices for learning rate and batch size, and comparing it to validation accuracy.
Revised: Figure 5 demonstrates this phenomenon by computing the Pearson correlation between the weights of converged warm-started models and their initialization weights, again across various choices for learning rate and batch size, and comparing it to validation accuracy.

Original: Models that generalize well have little correlation with their initialization—there is a trend downward in accuracy with increasing correlation—suggesting that they have forgotten what was learned in the ﬁrst round of training.
Revised: Models that generalize well have little correlation with their initialization—there is a trend downward in accuracy with increasing correlation—suggesting that they have forgotten what was learned in the ﬁrst round of training.

Original: Conversely, a similar plot for logistic regression shows no such relationship, and some of the best models have large correlations.
Revised: Conversely, a similar plot for logistic regression shows no such relationship, and some of the best models have large correlations.

Original: [Figure]
Revised: [Figure]

Original: V a li d a t i o n A cc u r a c y
Revised: V a li d a t i o n A cc u r a c y

Original: [Figure]
Revised: [Figure]

Original: 3. H OW QUICKLY IS GENERALIZATION DAMAGED ?
Revised: 3. H OW QUICKLY IS GENERALIZATION DAMAGED ?

Original: One of the surprising results of our investigation is how little training is necessary to damage the validation performance of the warm-started model.
Revised: One of the surprising results of our investigation is how little training is necessary to damage the validation performance of the warm-started model.

Original: Our hope was that warm-starting success might be achieved by switching from the 50% to 100% phase before the ﬁrst phase of training was completed .
Revised: Our hope was that warm-starting success might be achieved by switching from the 50% to 100% phase before the ﬁrst phase of training was completed .

Original: We did a search over switching times to try to identify whether there might be a “sweet spot” in which a partially-trained checkpoint might provide a good initialization.
Revised: We did a search over switching times to try to identify whether there might be a “sweet spot” in which a partially-trained checkpoint might provide a good initialization.

Original: We ﬁt a ResNet-18 model on 50% of the training data, as before, and checkpointed its parameters every ﬁve epochs.
Revised: We ﬁt a ResNet-18 model on 50% of the training data, as before, and checkpointed its parameters every ﬁve epochs.

Original: We then took each of these checkpointed models and used them as an initialization for training on 100% of those data.
Revised: We then took each of these checkpointed models and used them as an initialization for training on 100% of those data.

Original: As shown in Figure 7, generalization is damaged even when initializing from parameters obtained by training on incomplete data for only a few epochs.
Revised: As shown in Figure 7, generalization is damaged even when initializing from parameters obtained by training on incomplete data for only a few epochs.

Original: 3. A RE THERE DIFFERENCES IN THE GRADIENTS OF WARM - STARTED MODELS ?
Revised: 3. A RE THERE DIFFERENCES IN THE GRADIENTS OF WARM - STARTED MODELS ?

Original: We notice some interesting phenomena when examining the L 2 norm of the gradients as training progresses for a warm-started problem.
Revised: We notice some interesting phenomena when examining the L 2 norm of the gradients as training progresses for a warm-started problem.

Original: Figure 6 shows a visualization of average gradients during training for a ResNet on CIFAR-10.
Revised: Figure 6 shows a visualization of average gradients during training for a ResNet on CIFAR-10.

Original: On the left is the sequence of gradients during the ﬁrst phase of training.
Revised: On the left is the sequence of gradients during the ﬁrst phase of training.

Original: On the right is the second phase of training, where the magnitudes are shown separately for the two halves of the data set.
Revised: On the right is the second phase of training, where the magnitudes are shown separately for the two halves of the data set.

Original: It is unsurprising that the “old” data have zero gradients in the beginning, but it is surprising that the gradient magnitudes of the “new” data are higher even than at the random initialization.
Revised: It is unsurprising that the “old” data have zero gradients in the beginning, but it is surprising that the gradient magnitudes of the “new” data are higher even than at the random initialization.

Revised: This suggests they induce large changes to the model even though it has already been ﬁtted on samples from the same distribution.

Original: Moreover, the average magnitudes are very slow to converge; the new data have consistently larger gradients.
Revised: Moreover, the average magnitudes are very slow to converge; the new data have consistently larger gradients.

Original: 3. I S REGULARIZATION HELPFUL ?
Revised: 3. I S REGULARIZATION HELPFUL ?

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: The phenomenon of larger gradients induced by the addition of new data suggests that warmstarted models might beneﬁt from regularization.
Revised: The phenomenon of larger gradients induced by the addition of new data suggests that warmstarted models might beneﬁt from regularization.

Original: Here we investigate three different approaches to regularization: 1) basic L 2 weight penalties (Krogh and Hertz, 1992), 2) conﬁdencepenalized training (Pereyra et al., 2017) [Table] and 3) adversarial training (Szegedy et al., 2013).
Revised: Here we investigate three different approaches to regularization: 1) basic L 2 weight penalties (Krogh and Hertz, 1992), 2) conﬁdencepenalized training (Pereyra et al., 2017), and 3) adversarial training (Szegedy et al., 2014).

Original: We again take a ResNet ﬁtted to 50% of available training data and use its parameters to warm-start learning on 100% of those data.
Revised: We again take a ResNet ﬁtted to 50% of available training data and use its parameters to warm-start learning on 100% of those data.

Original: Regularization is applied in both rounds of training.
Revised: Regularization is applied in both rounds of training.

Original: Table 2 shows the results of these experiments.
Revised: Table 2 shows the results of these experiments.

Original: Regularization often helps, but it does not resolve the generalization gap created by warm-starting.
Revised: Regularization often helps, but it does not resolve the generalization gap created by warm-starting.

Original: We regularize both stages using the regularization scheme and penalty size indicated in Table 2.
Revised: We regularize both stages using the regularization scheme and penalty size indicated in Table 2.

Original: Still, applying the same regularization to randomly-initialized models always produces a better-generalizing classiﬁer.
Revised: Still, applying the same regularization to randomly-initialized models always produces a better-generalizing classiﬁer.

Original: 3. C AN WE WARM - START SOME LAYERS BUT NOT OTHERS ?
Revised: 3. C AN WE WARM - START SOME LAYERS BUT NOT OTHERS ?

Original: A common practice in deep learning is to train on one task, then continue training only the last network layer when new data become available (Mou et al., 2016; Karpathy and Johnson, 2019).
Revised: A common practice in deep learning is to train on one task, then continue training only the last network layer when new data become available (Mou et al., 2016; Karpathy and Johnson, 2019).

Original: This subsection investigates how performance is affected when we train a model on 50% of data, then use that initialization to retrain only the last layer of the network.
Revised: This subsection investigates how performance is affected when we train a model on 50% of data, then use that initialization to retrain only the last layer of the network.

Original: As the gradient of the last layer affects all earlier layers during training, it is one possible culprit for the vast gradient magnitude differences in Section 3.3.
Revised: As the gradient of the last layer affects all earlier layers during training, it is one possible culprit for the vast gradient magnitude differences in Section 3.3.

Original: We examine this hypothesis in two ways, as shown in Table 4.
Revised: We examine this hypothesis in two ways, as shown in Table 4.

Original: First we ran experiments that ﬁxed all parameters but the last layer to their pre-trained values and then only trained the last layer in the second phase (LL).
Revised: First we ran experiments that ﬁxed all parameters but the last layer to their pre-trained values and then only trained the last layer in the second phase (LL).

Original: Second we extend this experiment to a third phase in which the rest of the network was trained after allowing the last layer to converge to something reasonable (LL+WS).
Revised: Second we extend this experiment to a third phase in which the rest of the network was trained after allowing the last layer to converge to something reasonable (LL+WS).

Original: While training only the last layer from the warm-started initialization is typically worse than training all parameters (WS), some gains can be had by training the entire network after having only trained the last layer.
Revised: While training only the last layer from the warm-started initialization is typically worse than training all parameters (WS), some gains can be had by training the entire network after having only trained the last layer.

Original: That is, the last layer alone does not seem to be sufﬁcient to explain this generalization gap
Revised: That is, the last layer alone does not seem to be sufﬁcient to explain this generalization gap

Original: 3. I S THIS A PROBLEM WITH MOMENTUM OR OTHER PERSISTENT QUANTITIES DURING TRAINING ?
Revised: 3. I S THIS A PROBLEM WITH MOMENTUM OR OTHER PERSISTENT QUANTITIES DURING TRAINING ?

Original: One choice we have when using Adam or other adaptive methods involves whether or not to reset the optimization parameters after the ﬁrst round of training.
Revised: One choice we have when using Adam or other adaptive methods involves whether or not to reset the optimization parameters after the ﬁrst round of training.

Original: Can any part of this generalization gap be explained by a need to reinitialize hyperparameters or to use those that Adam has identiﬁed after training for a while?
Revised: Can any part of this generalization gap be explained by a need to reinitialize hyperparameters or to use those that Adam has identiﬁed after training for a while?

Original: The answer appears to be no : allowing hyperparameter settings to follow from the ﬁrst round of training to the second does not appear to affect out-of-sample performance of the ﬁnal solution.
Revised: The answer appears to be no : allowing hyperparameter settings to follow from the ﬁrst round of training to the second does not appear to affect out-of-sample performance of the ﬁnal solution.

Original: This procedure produces results that are very similar to Adam in Table 1, which reinitializes the optimizer before warm-start training.
Revised: This procedure produces results that are very similar to Adam in Table 1, which reinitializes the optimizer before warm-start training.

Original: 3. C AN WE JUST ADD NOISE ?
Revised: 3. C AN WE JUST ADD NOISE ?

Original: One idea for overcoming the warm-start problem is to add noise to the warm-started parameters, potentially jostling the network out of a poor initialization.
Revised: One idea for overcoming the warm-start problem is to add noise to the warm-started parameters, potentially jostling the network out of a poor initialization.

Original: The motivation for this is to have a “partially random” initialization.
Revised: The motivation for this is to have a “partially random” initialization.

Original: While adding noise of increased variance does help generalization, warm-started models are still lower-performing than randomly-initialized networks.
Revised: While adding noise of increased variance does help generalization, warm-started models are still lower-performing than randomly-initialized networks.

Original: Even worse, the train time for noised, warm-started models is proportional to the variance of the added noise.
Revised: Even worse, the train time for noised, warm-started models is proportional to the variance of the added noise.

Original: The best-generalizing noised warm-started networks actually take longer to ﬁt than freshly-initialized models (Table 3).
Revised: The best-generalizing noised warm-started networks actually take longer to ﬁt than freshly-initialized models (Table 3).

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 3. I S THIS A CASE OF CATASTROPHIC FORGETTING ?
Revised: 3. I S THIS A CASE OF CATASTROPHIC FORGETTING ?

Original: Warm starting is conceptually similar to continual learning.
Revised: Warm starting is conceptually similar to continual learning.

Original: In that framework, an agent is given learning tasks sequentially, and the goal is to become good at the current task while avoiding “catastrophic forgetting,”
Revised: In that framework, an agent is given learning tasks sequentially, and the goal is to become good at the current task while avoiding “catastrophic forgetting,”

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e., losing performance on previous tasks.
Revised: e., losing performance on previous tasks.

Original: One hypothesis to consider is whether the warm-start phenomenon is simply a case of such catastrophic forgetting.
Revised: One hypothesis to consider is whether the warm-start phenomenon is simply a case of such catastrophic forgetting.

Original: We can examine this hypothesis by trying to ﬁx the warm-start problem using a technique commonly used to prevent catastrophic forgetting.
Revised: We can examine this hypothesis by trying to ﬁx the warm-start problem using a technique commonly used to prevent catastrophic forgetting.

Original: One such approach is Elastic
Revised: One such approach is Elastic

Original: Weight Consolidation (EWC) (Kirkpatrick et al., 2017), which adds a regularization penalty that encourages avoiding updating weights that are important for previously-learned tasks.
Revised: Weight Consolidation (EWC) (Kirkpatrick et al., 2017), which adds a regularization penalty that encourages avoiding updating weights that are important for previously-learned tasks.

Original: [Table]
Revised: [Table]

Original: However, in the warm-start problem, each round of training adds data to what was available in the previous round.
Revised: However, in the warm-start problem, each round of training adds data to what was available in the previous round.

Original: As described here, these data are often from the same distribution as before.
Revised: As described here, these data are often from the same distribution as before.

Original: Because a model that works well on the second task of the warm-start problem also works well on the ﬁrst, there is no reason to avoid updating important parameters.
Revised: Because a model that works well on the second task of the warm-start problem also works well on the ﬁrst, there is no reason to avoid updating important parameters.

Original: Accordingly, as shown in Table 5, including the
Revised: Accordingly, as shown in Table 5, including the

Original: EWC penalty in warm-started network training actually damages performance more than not including it at all.
Revised: EWC penalty in warm-started network training actually damages performance more than not including it at all.

Original:  D ISCUSSION AND R ESEARCH S URROUNDING THE W ARM S TART P ROBLEM
Revised:  D ISCUSSION AND R ESEARCH S URROUNDING THE W ARM S TART P ROBLEM

Original: Warm-starting and online learning are well understood for convex models like linear classiﬁers (Chu et al., 2015) and SVMs (DeCoste and Wagstaff, 2000; Wen et al., 2017).
Revised: Warm-starting and online learning are well understood for convex models like linear classiﬁers (Chu et al., 2015) and SVMs (DeCoste and Wagstaff, 2000; Wen et al., 2017).

Original: However, it does not appear that generally applicable techniques exist for deep neural networks that do not damage generalization, and so models are typically retrained from scratch, e.g., Sener and Savarese (2018); Shyam et al.
Revised: However, it does not appear that generally applicable techniques exist for deep neural networks that do not damage generalization, and so models are typically retrained from scratch, e.g., Sener and Savarese (2018); Shyam et al.

Original: There has been a variety of work in closely related areas, however.
Revised: There has been a variety of work in closely related areas, however.

Original: For example, in analyzing “critical learning periods,” researchers show that a network initially trained on blurry images then on sharp images is unable to perform as well as one trained from scratch on sharp images, drawing a parallel between human vision and computer vision (Achille et al., 2018).
Revised: For example, in analyzing “critical learning periods,” researchers show that a network initially trained on blurry images then on sharp images is unable to perform as well as one trained from scratch on sharp images, drawing a parallel between human vision and computer vision (Achille et al., 2018).

Original: In this article, we show that this phenomenon occurs more generally and that test performance is damaged even when ﬁrst and second datasets are drawn from identical distributions.
Revised: In this article, we show that this phenomenon occurs more generally and that test performance is damaged even when ﬁrst and second datasets are drawn from identical distributions.

Original: Initialization.
Revised: Initialization.

Original: The problem of warm starting is closely related to the rich literature on initialization of neural network training “from scratch”.
Revised: The problem of warm starting is closely related to the rich literature on initialization of neural network training “from scratch”.

Original: Indeed, new insights into what makes an effective initialization have been critical to the revival of neural networks as machine learning models.
Revised: Indeed, new insights into what makes an effective initialization have been critical to the revival of neural networks as machine learning models.

Original: While there have been several proposed methods for initialization (Sutskever et al., 2013; Srivastava et al., 2015; Erhan et al., 2010; Glorot and Bengio, 2010; He et al., 2015), this body of literature primarily concerns itself with initializations that are high-quality in the sense that they allow quick and reliable training error minimization.
Revised: While there have been several proposed methods for initialization (Sutskever et al., 2013; Srivastava et al., 2015; Erhan et al., 2010; Glorot and Bengio, 2010; He et al., 2015), this body of literature primarily concerns itself with initializations that are high-quality in the sense that they allow quick and reliable training error minimization.

Original: Work relating initialization to generalization suggests that networks whose weights have moved far from their initialization are less likely to generalize well compared with ones that have remained relatively nearby (Nagarajan and Kolter, 2019).
Revised: Work relating initialization to generalization suggests that networks whose weights have moved far from their initialization are less likely to generalize well compared with ones that have remained relatively nearby (Nagarajan and Kolter, 2017).

Original: While this makes sense when learning a new problem from scratch, warm-started networks that have less in common with their initializations seem to generalize better than those that have more (Figure 5).
Revised: While this makes sense when learning a new problem from scratch, warm-started networks that have less in common with their initializations seem to generalize better than those that have more (Figure 5).

Original: So while it is not surprising that there exist initializations that generalize poorly, it is surprising that the initializations discussed in Section 2 are in that class.
Revised: So while it is not surprising that there exist initializations that generalize poorly, it is surprising that the initializations discussed in Section 1 are in that class.

Original: Generalization.
Revised: Generalization.

Original: The warm start problem is fundamentally about generalization performance, which has been extensively studied both theoretically and empirically within the context of deep learning.
Revised: The warm start problem is fundamentally about generalization performance, which has been extensively studied both theoretically and empirically within the context of deep learning.

Original: These articles have investigated generalization by studying classiﬁer margin (Bartlett et al., 2017; Wei et al., 2018), loss geometry (Hochreiter and Schmidhuber, 1997;
Revised: These articles have investigated generalization by studying classiﬁer margin (Bartlett et al., 2017; Wei et al., 2018), loss geometry (Hochreiter and Schmidhuber, 1997;

Original: Keskar et al., 2016), and measurements of complexity (Zhang et al., 2017; Liang et al., 2017), sensitivity (Novak et al., 2018), or compressiblity (Zhou et al., 2019).
Revised: Keskar et al., 2016; Li et al., 2018a), and measurements of complexity (Zhang et al., 2017; Liang et al., 2017), sensitivity (Novak et al., 2018), or compressiblity (Zhou et al., 2019).

Original: These approaches can be seen as attempting to measure the intricacy of the hypothesis learned by the network.
Revised: These approaches can be seen as attempting to measure the intricacy of the hypothesis learned by the network.

Original: If two models are both consistent for the same training data, the one with the less complicated concept is more likely to generalize well.
Revised: If two models are both consistent for the same training data, the one with the less complicated concept is more likely to generalize well.

Original: We know that networks trained with SGD are implicitly regularized (Li et al., 2017; Gunasekar et al., 2017), suggesting that standard training of neural networks incidentally ﬁnds low-complexity solutions.
Revised: We know that networks trained with SGD are implicitly regularized (Li et al., 2018b; Gunasekar et al., 2017), suggesting that standard training of neural networks incidentally ﬁnds low-complexity solutions.

Original: It’s possible, then, that the initial round of training disqualiﬁes solutions that would most naturally explain the general problem of interest.
Revised: It’s possible, then, that the initial round of training disqualiﬁes solutions that would most naturally explain the general problem of interest.

Original: If so, it would imply that the warm-start problem is a pathology of SGD.
Revised: If so, it would imply that the warm-start problem is a pathology of SGD.

Original: Pre-training.
Revised: Pre-training.

Original: As previously discussed, the warm-start problem is very similar to the idea of unsupervised and supervised pre-training (Li et al., 2019; Bengio, 2011; Erhan et al., 2010; Bengio et al., 2007).
Revised: As previously discussed, the warm-start problem is very similar to the idea of unsupervised and supervised pre-training (Li et al., 2019; Bengio, 2011; Erhan et al., 2010; Bengio et al., 2007).

Original: Under that paradigm, learning where limited labeled data are available is aided by ﬁrst training on related data.
Revised: Under that paradigm, learning where limited labeled data are available is aided by ﬁrst training on related data.

Original: The warm start problem, however, is not about limited labeled data in the second round of training.
Revised: The warm start problem, however, is not about limited labeled data in the second round of training.

Original: Instead, the goal of warm starting is to hasten the time required to ﬁt a neural network by initializing using a similar supervised problem without damaging generalization.
Revised: Instead, the goal of warm starting is to hasten the time required to ﬁt a neural network by initializing using a similar supervised problem without damaging generalization.

Original: Our results suggest that while warm-starting is beneﬁcial when labeled data are limited, it actually damages generalization to warm-start in rich-data situations.
Revised: Our results suggest that while warm-starting is beneﬁcial when labeled data are limited, it actually damages generalization to warm-start in rich-data situations.

Original: This is somewhat opposed to conventional intuition, which suggests that pre-training is helpful when there are limited data available, but that it is not harmful when data are not limited (Karpathy and Johnson, 2019).
Revised: This is somewhat opposed to conventional intuition, which suggests that pre-training is helpful when there are limited data available, but that it is not harmful when data are not limited (Karpathy and Johnson, 2019).

Original: Concluding thoughts.
Revised: Concluding thoughts.

Original: This article presented challenges of warm-starting neural network training.
Revised: This article presented challenges of warm-starting neural network training.

Original: While this is a problem that the community seems somewhat aware of anecdotally, it does not seem to have been directly studied.
Revised: While this is a problem that the community seems somewhat aware of anecdotally, it does not seem to have been directly studied.

Original: We believe that this is a major problem in important real-life tasks for which neural networks are used, and it speaks directly to the resources consumed by training such models.
Revised: We believe that this is a major problem in important real-life tasks for which neural networks are used, and it speaks directly to the resources consumed by training such models.

Original: We also observe that while the presented experiments show strong evidence that the warm start problem is not something that can be easily overcome with current optimization tools, there are still many avenues to investigate.
Revised: We also observe that while the presented experiments show strong evidence that the warm start problem is not something that can be easily overcome with current optimization tools, there are still many avenues to investigate.

Original: For example, Polyak averaging (Polyak and Juditsky, 1992), AdaGrad (Duchi et al., 2011), and
Revised: For example, Polyak averaging (Polyak and Juditsky, 1992), AdaGrad (Duchi et al., 2011), and

Original: KFAC (Martens and Grosse, 2015) are techniques that ﬁnd favor on certain problems and architectures and might make warm-starting possible.
Revised: KFAC (Martens and Grosse, 2015) are techniques that ﬁnd favor on certain problems and architectures and might make warm-starting possible.

Original: Additionally, we have focused on widely-studied image data sets; we leave similar analyses of other tasks such as natural language processing and molecular function prediction to future work.  
Revised: Additionally, we have focused on widely-studied image data sets; we leave similar analyses of other tasks such as natural language processing and molecular function prediction to future work.  A PPENDIX T ABLES [Table] [Table]

Revised: validation performance.

Revised: [Figure]

Original:  A PPENDIX 
Revised:  A PPENDIX F IGURES

Original: F IGURES

Original: [Figure] [Figure]
Revised: [Figure] [Figure]