Original: Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks
Revised: Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: A striking observation about iterative magnitude pruning (IMP; Frankle et al.
Revised: A striking observation about iterative magnitude pruning (IMP; Frankle et al.

Original: 10)is that—after just a few hundred steps of dense training—the method can ﬁnd asparse sub-network that can be trained to the same accuracy as the dense network.
Revised: 6)is that—after just a few hundred steps of dense training—the method can ﬁnd asparse sub-network that can be trained to the same accuracy as the dense network.

Original: However, the same does not hold at step 0, i.e., random initialization.
Revised: However, the same does not hold at step 0, i.e., random initialization.

Original: In thiswork, we seek to understand how this early phase of pre-training leads to a goodinitialization for IMP both through the lens of the data distribution and the losslandscape geometry.
Revised: In thiswork, we seek to understand how this early phase of pre-training leads to a goodinitialization for IMP both through the lens of the data distribution and the losslandscape geometry.

Original: Empirically we observe that, holding the number of pretraining iterations constant, training on a small fraction of (randomly chosen) datasufﬁces to obtain an equally good initialization for IMP.
Revised: Empirically we observe that, holding the number of pretraining iterations constant, training on a small fraction of (randomly chosen) datasufﬁces to obtain an equally good initialization for IMP.

Original: We additionally observethat by pre-training only on “easy” training data we can decrease the number ofsteps necessary to ﬁnd a good initialization for IMP compared to training on thefull dataset or a randomly chosen subset.
Revised: We additionally observethat by pre-training only on “easy” training data we can decrease the number ofsteps necessary to ﬁnd a good initialization for IMP compared to training on thefull dataset or a randomly chosen subset.

Original: Finally, we identify novel propertiesof the loss landscape of dense networks that are predictive of IMP performance,showing in particular that more examples being linearly mode connected in thedense network correlates well with good initializations for IMP.
Revised: Finally, we identify novel propertiesof the loss landscape of dense networks that are predictive of IMP performance,showing in particular that more examples being linearly mode connected in thedense network correlates well with good initializations for IMP.

Original: Combined, theseresults provide new insight into the role played by the early phase training in IMP.
Revised: Combined, theseresults provide new insight into the role played by the early phase training in IMP.

Original:  Introduction
Revised:  Introduction

Original: Modern deep neural networks are often trained in the massively over-parameterized regime.
Revised: Modern deep neural networks are often trained in the massively over-parameterized regime.

Original: Thoughthese networks can eventually be pruned, quantized, or distilled into smaller networks, the resourcesrequired for the initial over-parameterized training poses a challenge to the democratization andsustainability of AI.
Revised: Thoughthese networks can eventually be pruned, quantized, or distilled into smaller networks, the resourcesrequired for the initial over-parameterized training poses a challenge to the democratization andsustainability of AI.

Original: This raises a fundamental question: under what circumstances can we efﬁcientlytrain sparse networks?
Revised: This raises a fundamental question: under what circumstances can we efﬁcientlytrain sparse networks?

Original: Recent work on the lottery ticket hypothesis [9, 10] has shown that, after just afew hundred steps of pre-training, a dense network contains a sparse sub-network that can be trainedwithout any loss in performance.
Revised: Recent work on the lottery ticket hypothesis [5, 6] has shown that, after just afew hundred steps of pre-training, a dense network contains a sparse sub-network that can be trainedwithout any loss in performance.

Original: Finding this sparse sub-network currently requires multiple roundsof training to convergence, pruning and rewinding to the pre-train point, a procedure termed iterativemagnitude pruning (IMP, Figure 1; [9, 10]), Remarkably, even after all these rounds of training, wedo not ﬁnd trainable sparse sub-networks if we rewind to the random initialization; the ﬁrst fewhundred steps of dense network training is essential for ﬁnding sparse networks through IMP.
Revised: Finding this sparse sub-network currently requires multiple roundsof training to convergence, pruning and rewinding to the pre-train point, a procedure termed iterativemagnitude pruning (IMP, Figure 1; [5, 6]), Remarkably, even after all these rounds of training, we donot ﬁnd trainable sparse sub-networks if we rewind to the random initialization; the ﬁrst few hundredsteps of dense network training is essential for ﬁnding sparse networks through IMP.

Original: Inthis work, we seek to understand this very short but critical phase of pre-training.
Revised: In this work, weseek to understand this very short but critical phase of pre-training.

Original: In particular, weinvestigate the effect of training data and number of steps used during pre-training on the accuracyachieved by IMP.
Revised: In particular, we investigate theeffect of training data and number of steps used during pre-training on the accuracy achieved by IMP.

Original: We also explain how certain properties of the loss landscape allow us to predictwhether we will ﬁnd a matching initialization (i.e., an initialization from which IMP “succeeds” inﬁnding a subnetwork that can match the accuracy of the unpruned network, to be formalized later)
Revised: We also explain how certain properties of the loss landscape allow us to predict whether we will ﬁnda matching initialization (i.e., an initialization from which IMP “succeeds” in ﬁnding a subnetworkthat can match the accuracy of the unpruned network, to be formalized later).

Original: IMP proceeds in three phases: an initial pre-training phase where (1) the dense weights are trainedfor a few hundred steps, followed by (2) the resource-intensive mask search phase , during which we iteratively ﬁnd a sparse mask by training the network and rewinding the weights repeatedly, ﬁnally,[Figure]
Revised: IMP proceeds in three phases: an initial pre-training phase where (1) the dense weights are trainedfor a few hundred steps, followed by (2) the resource-intensive mask search phase , during which weiteratively ﬁnd a sparse mask by training the network and rewinding the weights repeatedly, ﬁnally,

Original: Iteration k
Revised: Iteration k

Revised: Iteration k

Original: Iteration T
Revised: Iteration T

Original: P r un e a n d R e w i n d
Revised: P r un e a n d R e w i n d

Original: Iteration
Revised: Iteration

Original: Iteration T
Revised: Iteration T

Original: Iteration k
Revised: Iteration k

Original: Iteration T
Revised: Iteration T

Original: Pruning 
Revised: Pruning Round

Original: [Figure]
Revised: Final Mask

Original: P r un e a n d R e w i n d
Revised: P r un e a n d R e w i n d

Original: Phase 1: Pre- [Figure]
Revised: Phase 1: Pre-training

Original: Phase 3:
Revised: Phase 3:

Original: Sparse Training
Revised: Sparse Training

Original: P h a s e 2 :
Revised: P h a s e 2 :

Original: M a s k S e a r c h
Revised: M a s k S e a r c h

Original: [Figure]

Original: sparse subnetwork ending with (3) the sparse training phase , when the sparse masked network is trained.
Revised: Train Loss Barrier ending with (3) the sparse training phase , when the sparse masked network is trained.

Original: The weightslearned after pre-training phase—which serve as the initialization for mask search phase—are referredto as the pre-trained initialization (see Figure 1).
Revised: The weightslearned after pre-training phase—which serve as the initialization for mask search phase—are referredto as the pre-trained initialization (see Figure 1).

Original: In this work, we focus on studying the pre-training phase.
Revised: In this work, we focus on studying the pre-training phase.

Original: To probe sufﬁcient information needed toarrive at a matching initialization for IMP, we modify the pre-training phase by training with differentpruned datasets and varying the number of pre-training steps; we then compare the performanceof the resulting sub-networks across a range of sparsities.
Revised: To probe sufﬁcient information needed toarrive at a matching initialization for IMP, we modify the pre-training phase by training with differentpruned datasets and varying the number of pre-training steps; we then compare the performanceof the resulting sub-networks across a range of sparsities.

Original: Following Paul et al.
Revised: Following Paul et al.

Original: [23], we prunethe datasets both randomly and according to EL2N scores, which estimate example difﬁculty bymeasuring early training performance in an ensemble of networks.
Revised: [14], we prunethe datasets both randomly and according to EL2N scores, which estimate example difﬁculty bymeasuring early training performance in an ensemble of networks.

Original: We next turn to characterizingthe loss landscape properties of matching initialization for IMP.
Revised: We next turn to characterizingthe loss landscape properties of matching initialization for IMP.

Original: Frankle et al.
Revised: Frankle et al.

Original: [10] investigate therelationship between successful initializations for IMP and linear mode connectivity on the sparseinitialization for the entire dataset; here, we investigate this relationship on the dense initialization ona per-example basis.
Revised: [6] investigate therelationship between successful initializations for IMP and linear mode connectivity on the sparseinitialization for the entire dataset; here, we investigate this relationship on the dense initialization ona per-example basis.

Original: This enables us to identify signatures of the dense loss landscape correlatedwith a matching initialization.
Revised: This enables us to identify signatures of the dense loss landscape correlatedwith a matching initialization.

Original: Thinking more broadly, we explore the relationship between sparsetrainability and training stability.
Revised: Thinking more broadly, we explore the relationship between sparsetrainability and training stability.

Original: Gilmer et al.
Revised: Gilmer et al.

Original: [13] demonstrate that, in hyperparameter regimes inwhich early training is unstable, learning rate warmup helps stabilize training.
Revised: [9] demonstrate that, in hyperparameter regimes inwhich early training is unstable, learning rate warmup helps stabilize training.

Original: We build on this byinvestigating if the same pruned datasets which decreased the amount of pre-training required to ﬁnda matching initialization for IMP also decrease the required duration of learning rate warmup.
Revised: We build on this byinvestigating if the same pruned datasets which decreased the amount of pre-training required to ﬁnda matching initialization for IMP also decrease the required duration of learning rate warmup.

Original: Overall, this empirical analysis provides new insight into the important and mysterious role thatpre-training the dense network plays in IMP.
Revised: Overall, this empirical analysis provides new insight into the important and mysterious role thatpre-training the dense network plays in IMP.

Original: Intriguingly, ﬁnding a matching initialization for IMPis an example of a problem in which the dominant paradigm of “more data, more training” is notoptimal.
Revised: Intriguingly, ﬁnding a matching initialization for IMPis an example of a problem in which the dominant paradigm of “more data, more training” is notoptimal.

Original: Indeed, by carefully choosing the training examples, we can not only use a very smallsubset of the data but also reduce the training time required to solve this problem.
Revised: Indeed, by carefully choosing the training examples, we can not only use a very smallsubset of the data but also reduce the training time required to solve this problem.

Original: From a scientiﬁcperspective, this suggests that different phases of training play different roles in the optimizationprocess.
Revised: From a scientiﬁcperspective, this suggests that different phases of training play different roles in the optimizationprocess.

Original: Identifying and characterizing these phases will not only lead to a deeper understandingof the optimization of deep neural networks but also allow us to design training strategies that areoptimal for each phase.
Revised: Identifying and characterizing these phases will not only lead to a deeper understandingof the optimization of deep neural networks but also allow us to design training strategies that areoptimal for each phase.

Original: From the practical standpoint, data loading is often an expensive part of thetraining process and can be a bottleneck especially early in training.
Revised: From the practical standpoint, data loading is often an expensive part of thetraining process and can be a bottleneck especially early in training.

Original: By identifying the essentialsubset for this phase of training, our work may enable strategies to eliminate this bottleneck.
Revised: By identifying the essentialsubset for this phase of training, our work may enable strategies to eliminate this bottleneck.

Original: Contributions.
Revised: Contributions.

Original: We ﬁnd empirical evidence for the following statements:
Revised: We ﬁnd empirical evidence for the following statements:

Original: • In the pre-training phase, only a small fraction of the data is required to ﬁnd a matchinginitialization for IMP: On standard benchmarks, across all sparsity levels we evaluated, we ﬁndthat we can match accuracy by training on a small fraction of all of the available training data,selected randomly.
Revised: • In the pre-training phase, only a small fraction of the data is required to ﬁnd a matchinginitialization for IMP: On standard benchmarks, across all sparsity levels we evaluated, we ﬁndthat we can match accuracy by training on a small fraction of all of the available training data,selected randomly.

Original: (As we vary the amount of training data in pre-training phase, the numberof training iterations is held ﬁxed.)
Revised: (As we vary the amount of training data in pre-training phase, the numberof training iterations is held ﬁxed.)

Original: Note that this observation changes if random label noise is introduced, in which case it becomes important to select easy (small EL2N score) examples.
Revised: Note that this observation changes if random label noise isintroduced, in which case it becomes important to select easy (small EL2N score) examples.

Original: • The length of the pre-training phase can be reduced if we train only on the easiest examples: Informally, training on a small subset of “easy-to-learn” training examples produces a betterrewinding point than training on all data for the same number of iterations;
Revised: • The length of the pre-training phase can be reduced if we train only on the easiest examples:Informally, training on a small subset of “easy-to-learn” training examples produces a betterrewinding point than training on all data for the same number of iterations;

Original: • The quality of a pre-trained initialization for IMP correlates with more examples beinglinearly mode connected in the dense network: This result complements the empirical evidenceproduced by Frankle et al.
Revised: • The quality of a pre-trained initialization for IMP correlates with more examples beinglinearly mode connected in the dense network: This result complements the empirical evidenceproduced by Frankle et al.

Original: [10] connecting linear mode connectivity and the performance of IMP.• Training on easy data, which reduces the amount of pre-training required to ﬁnd a matchinginitialization for IMP, does not reduce the amount of training required to stablize the networkvia learning rate warmup ([13]): The role played by data during IMP pre-training is thus differentthan that during learning rate warmup.
Revised: [6] connecting linear mode connectivity and the performance of IMP.• Training on easy data, which reduces the amount of pre-training required to ﬁnd a matchinginitialization for IMP, does not reduce the amount of training required to stablize the networkvia learning rate warmup: Running learning rate warmup on easy data cannot replicate thewarmup effect on the top loss Hessian eigenvalue ([9]), but a small fraction of medium-hard tohard data sufﬁces.

Original:  Background, Methods, and Related Work
Revised:  Background, Methods, and Related Work

Original: We consider standard neural network training on image classiﬁcation.
Revised: We consider standard neural network training on image classiﬁcation.

Original: Let S = { ( x n , y n ) } Nn =1denote training data, let w t 2 R D denote model parameters (weights) of the neural network, andlet w 1 , w 2 , . .
Revised: Let S = { ( x n , y n ) } Nn =1denote training data, let w t 2 R D denote model parameters (weights) of the neural network, andlet w 1 , w 2 , . .

Original: . be the iterates of (some variant) of SGD, minimizing the training loss , i.e., averagecross-entropy loss over the training data.
Revised: . be the iterates of (some variant) of SGD, minimizing the training loss , i.e., averagecross-entropy loss over the training data.

Original: For a given training example x n , let f ( w , x ) 2 R K denotethe logit outputs of the network for weights w and p ( w , x )
Revised: For a given training example x n , let f ( w, x ) 2 R K denotethe logit outputs of the network for weights w and p ( w, x )

Original: =
Revised: =

Original:  ( f ( w , x )) be the probability vectorreturned by passing the logits through the softmax operation  .
Revised:  ( f ( w, x )) be the probability vectorreturned by passing the logits through the softmax operation  .

Original: By the loss (error) landscape , wemean the training loss (error), viewed as a function of the parameters.
Revised: By the loss (error) landscape , wemean the training loss (error), viewed as a function of the parameters.

Original: By training and test error , wemean the average 0-1 classiﬁcation loss.
Revised: By training and test error , wemean the average 0-1 classiﬁcation loss.

Original: Lottery ticket subnetworks.
Revised: Lottery ticket subnetworks.

Original: The lottery ticket hypothesis [9] states that any standard neural network“contains [at initialization] a subnetwork that is initialized such that—when trained in isolation—itcan match the test accuracy of the original network after training for at most the same number ofiterations.”
Revised: The lottery ticket hypothesis [5] states that any standard neural network“contains [at initialization] a subnetwork that is initialized such that—when trained in isolation—itcan match the test accuracy of the original network after training for at most the same number ofiterations.”

Original: Although such matching subnetworks (those that can train to completion on their ownand reach full accuracy by following the same procedure as the unpruned network) are not known toexist in general at random initialization, they have been shown to exist after pre-training the densenetwork for a short amount of time (Phase 1 in Figure 1) before pruning [2, 10, 17, 25, 27].
Revised: Although such matching subnetworks (those that can train to completion on their ownand reach full accuracy by following the same procedure as the unpruned network) are not known toexist in general at random initialization, they have been shown to exist after pre-training the densenetwork for a short amount of time (Phase 1 in Figure 1) before pruning [1, 6, 12, 15, 16].

Original: Empirical evidence for this phenomenon comes via a procedure that ﬁnds such subnetworks retroactively after training the entire network.
Revised: Empirical evidence for this phenomenon comes via a procedure that ﬁnds such subnetworks retroactively after training the entire network.

Original: This procedure, called Iterative Magnitude Pruning [IMP;10] is based on standard iterative pruning procedures [14], and can be decomposed into three phases(Figure 1), outlined in Algorithm 1.
Revised: This procedure, called Iterative Magnitude Pruning [IMP;6] is based on standard iterative pruning procedures [10], and can be decomposed into three phases(Figure 1), outlined in Algorithm 1.

Original: 1: Create a network with randomly initialization w 0 2 R d . 2: Initialize pruning mask to m = 1 d . 3:Train w 0 for t r steps to w t r
Revised: 1: Create a network with randomly initialization w 0 2 R d . 2: Initialize pruning mask to m = 1 d . 3:

Original: 4: forn 2 { 1 , . . .
Revised: 4: for

Original: , N } do
Revised: do

Original: 5: Train the pruned network m  w t r to completion.
Revised: 5: Train the pruned network m  w t r to completion.

Original: (  is the element-wise product) 6: Prune the lowest magnitude 20% of weights after training.
Revised: the element-wise product) 6: Prune the lowest magnitude 20% of weights after training.

Original: Let m [ i ] =
Revised: Let m [ i ] =

Original: if the corresponding weight is pruned.
Revised: if the corresponding weight is pruned.

Original: 7: Train the ﬁnal network m  w t r .
Revised: 7: Train the ﬁnal network m  w t r .

Original: Measure its accuracy.
Revised: Measure its accuracy.

Original: Training This procedure reveals the accuracy of pre-training the dense network for t r iterations, pruning,and training the pruned network thereafter.
Revised: This procedure reveals the accuracy of pre-training the dense network for t r iterations, pruning,and training the pruned network thereafter.

Original: Phase 2 can be understood as an (expensive) oracle forchoosing weights to prune at t r .
Revised: Phase 2 can be understood as an (expensive) oracle forchoosing weights to prune at t r .

Original: Although IMP is too expensive to use as a practical way to speedup training, it provides a window into a possible minimal number of parameters and operationsnecessary to successfully train a network to completion in practice.
Revised: Although IMP is too expensive to use as a practical way to speedup training, it provides a window into a possible minimal number of parameters and operationsnecessary to successfully train a network to completion in practice.

Original: In our work, we extend this lineof thinking, pursuing the minimal amount of data necessary to ﬁnd and train these subnetworks.
Revised: In our work, we extend this lineof thinking, pursuing the minimal amount of data necessary to ﬁnd and train these subnetworks.

Original: Thisis especially tantalizing due to the potential positive interactions between sparsity and minimizingthe data necessary for training.
Revised: Thisis especially tantalizing due to the potential positive interactions between sparsity and minimizingthe data necessary for training.

Original: The result is a deeper inquiry into the minimal recipe for successfultraining and, thereby, into the fundamental nature of neural network learning in practice.
Revised: The result is a deeper inquiry into the minimal recipe for successfultraining and, thereby, into the fundamental nature of neural network learning in practice.

Original: In this respect, the closest work to ours is an experiment in a larger compendium by Frankle et al.
Revised: In this respect, the closest work to ours is an experiment in a larger compendium by Frankle et al.

Original: [11]showing that the standard pre-training phase could be replaced by a much longer self-supervised phase.
Revised: [7]showing that the standard pre-training phase could be replaced by a much longer self-supervised phase.

Original: Like our experiments in Section 3, that work aims to study what makes a matching initialization.
Revised: Like our experiments in Section 3, that work aims to study what makes a matching initialization.

Original: Ourapproach and ﬁndings are substantially different, however: we reduce the number of examples ratherthan changing the labels, and we show that, not only are a small set of examples (starting at ⇡ 2% )sufﬁcient for pre-training, but also that they make it possible to pre-train in fewer steps.
Revised: Ourapproach and ﬁndings are substantially different, however: we reduce the number of examples ratherthan changing the labels, and we show that—not only are a small set of examples (starting at ⇡ 2% )sufﬁcient for pre-training—but also that they make it possible to pre-train in fewer steps.

Original: There are many other ways to obtain pruned neural networks [e.g., 7, 14, 16, 20, 29].
Revised: There are many other ways to obtain pruned neural networks [e.g., 3, 10, 11, 13, 18].

Original: The distinctiveaspect of work on the lottery ticket hypothesis (and the one that makes it the right starting point forour inquiry) is that its goal is to uncover a minimal path from initialization to a trained network,regardless of the cost of doing so.
Revised: The distinctiveaspect of work on the lottery ticket hypothesis (and the one that makes it the right starting point forour inquiry) is that its goal is to uncover a minimal path from initialization to a trained network,regardless of the cost of doing so.

Original: The aforementioned procedures target real-world efﬁciency fortraining and/or inference.
Revised: The aforementioned procedures target real-world efﬁciency fortraining and/or inference.

Original: Linear Mode Connectivity and the Loss Barrier.
Revised: Linear Mode Connectivity and the Loss Barrier.

Original: We investigate the error landscape using theparent–child methodology and instability analysis of Frankle et al.
Revised: We investigate the error landscape using theparent–child methodology and instability analysis of Frankle et al.

Original: [10] and Fort et al.
Revised: [6] and Fort et al.

Original: Writingerr( w ) for the test error at the weights w , the (test) loss barrier between two networks w and w 0is sup ↵ 2 [0 , 1] [err( ↵ w + (1  ↵ ) w 0 )  ( ↵ err( w ) + (1  ↵ )err( w 0 ))] .
Revised: Writingerr( w ) for the test error at the weights w , the (test) loss barrier between two networks w and w 0is sup ↵ 2 [0 , 1] [err( ↵ w + (1  ↵ ) w 0 )  ( ↵ err( w ) + (1  ↵ )err( w 0 ))] .

Original: This quantity measures themaximum increase in loss above the average loss along the linear path connecting the two networkson the loss landscape.
Revised: This quantity measures themaximum increase in loss above the average loss along the linear path connecting the two networkson the loss landscape.

Original: The loss barrier at iteration t (with parent weights w t ) is the loss barrier between the (children)weights w T and w 0 T , where w T and w 0 T are copies of w t trained to completion with the sameprocedure but different random seeds (minibatch order, GPU noise, data augmentation, etc.).
Revised: The loss barrier at iteration t (with parent weights w t ) is the loss barrier between the (children) weightsw T and w 0 T , where w T and w 0 T are copies of w t trained to completion with the same procedure butdifferent random seeds (minibatch order, GPU noise, data augmentation, etc.).

Original: See Figure 1 right for a visualization.
Revised: See Figure 1 right fora visualization.

Original: Empirically, the loss barrier is achieved near ↵ = 12 , and so wecompute it this way.
Revised: Empirically, the loss barrier is achieved near ↵ = 12 , and so we compute it this way.

Original: The onset of linear mode connectivity (LMC) is deﬁned to be the iteration ¯ t suchthat, for all t > ¯ t , the error barrier at t is zero.
Revised: The onset of linear mode connectivity (LMC) is deﬁned to be the iteration ¯ t such that, for all t > ¯ t ,the error barrier at t is zero.

Original: We follow Draxler et al.
Revised: We follow Draxler et al.

Original: [4], Garipov et al.
Revised: [2], Garipov et al.

Original: [12], and Frankle et al.
Revised: [8], and Frankle et al.

Original: [10] in considering the 0-1 loss barrier to be zero if it is less than 2%.
Revised: [6] inconsidering the 0-1 loss barrier to be zero if it is less than 2%.

Original: Similar to Paulet al.
Revised: Similar to Paul et al.

Original: [23], we also measure the error (0-1 loss) or cross-entropy loss barrier on individual trainingexamples to explore the loss landscape corresponding to different subpopulations.
Revised: [14], we alsomeasure the error (0-1 loss) or cross-entropy loss barrier on individual training examples to explorethe loss landscape corresponding to different subpopulations.

Original: Ranking training examples.
Revised: Ranking training examples.

Original: We deﬁne “easy/hard data” as the data that is ranked low/high, respectively, by the EL2N score introduced by Paul et al.
Revised: We deﬁne “easy/hard data” as the data that is ranked low/high, respectively, by the EL2N score introduced by Paul et al.

Original: EL2N scores depend on the margin early intraining, and, loosely speaking, higher average margin early in training means lower importance forgeneralization of the ﬁnal trained model.
Revised: EL2N scores depend on the margin early intraining, and, loosely speaking, higher average margin early in training means lower importance forgeneralization of the ﬁnal trained model.

Original: This connection to margin suggests that easy data is learnedﬁrst (has higher margin early in training, maintained throughout the rest of training).
Revised: This connection to margin suggests that easy data is learnedﬁrst (has higher margin early in training, maintained throughout the rest of training).

Original: EL2N scoreswere derived from the size of the loss gradient, and are thus are highly correlated with the magnitudeof the gradient.
Revised: EL2N scoreswere derived from the size of the loss gradient, and are thus are highly correlated with the magnitudeof the gradient.

Original: Deﬁnition 2.1 (EL2N Score)
Revised: Deﬁnition 2.1 (EL2N Score)

Original: The EL2N score of a training sample ( x , y ) at iteration t is deﬁned as
Revised: The EL2N score of a training sample ( x, y ) at iteration t is deﬁned as

Original: E k p ( w t , x )  y k 2 , where the expectation is taken over w t conditioned on the training data.
Revised: E k p ( w t , x )  y k 2 , where the expectation is taken over w t conditioned on the training data.

Original: In our experiments (Section 3), we vary the data that is accessible in the pre-training phase of IMPdeﬁned above.
Revised: In our experiments (Section 3), we vary the data that is accessible in the pre-training phase of IMPdeﬁned above.

Original: We either choose the data that we feed to the algorithm at random while preservingclass balance, or based on the EL2N scores.
Revised: We either choose the data that we feed to the algorithm at random while preservingclass balance, or based on the EL2N scores.

Original: Stabilizing early training via learning rate warmup.
Revised: Stabilizing early training via learning rate warmup.

Original: Learning rate warmup period can be seenas a form of pre-training, allowing one to eventually train at higher learning rates and larger batchsizes.
Revised: Learning rate warmup period can be seenas a form of pre-training, allowing one to eventually train at higher learning rates and larger batchsizes.

Original: Training with large batches and high learning rates is desirable in practice, as it allows for moreefﬁcient GPU utilization and may reduce the total number of updates needed to achieve the desiredaccuracy.
Revised: Training with large batches and high learning rates is desirable in practice, as it allows formore efﬁcient GPU utilization and may reduce the total number of updates needed to achieve thedesired accuracy.

Original: Gilmer et al.
Revised: Gilmer et al.

Original: [13] empirically observe that learning rate warmup essentially improvesthe optimization by allowing the initial optimization trajectory to navigate to “ﬂatter” optimizationlandscape, i.e., one with smaller highest loss Hessian eigenvalue.
Revised: [9] empirically observe that learning rate warmup essentially improvesthe optimization by allowing the initial optimization trajectory to navigate to “ﬂatter” optimizationlandscape, i.e., one with smaller highest loss Hessian eigenvalue.

Original:  The Role of Training Data Selection in Pre-Training
Revised:  The Role of Training Data Selection in Pre-Training

Original: As can be seen in Figure 2, when training sparse networks using IMP with rewind step t r = 0 , theﬁnal test accuracy of the sparse networks falls off rapidly with increasing sparsity.
Revised: As can be seen in Figure 2, when training sparse networks using IMP with rewind step t r = 0 , theﬁnal test accuracy of the sparse networks falls off rapidly with increasing sparsity.

Original: However, as weincrease the rewind step t r , the network performance improves across all sparsity levels and at arewind step t ⇤ , the network performs as well as or better than the dense network at high sparsities.
Revised: However, as weincrease the rewind step t r , the network performance improves across all sparsity levels and at arewind step t ⇤ , the network performs as well as or better than the dense network at high sparsities.

Original: Informally, training the dense network for t ⇤ steps creates a matching initialization for IMP.
Revised: Informally, training the dense network for t ⇤ steps creates a matching initialization for IMP.

Original: But whatdoes the network learn in these ﬁrst t ⇤ steps?
Revised: But whatdoes the network learn in these ﬁrst t ⇤ steps?

Original: In this section, we take the ﬁrst step towards answeringthis question by investigating which subsets of the training data are sufﬁcient for ﬁnding a matchinginitialization.
Revised: In this section, we take the ﬁrst step towards answeringthis question by investigating which subsets of the training data are sufﬁcient for ﬁnding a matchinginitialization.

Original: In order to compare networks trained on different subsets of data for different numbersof iterations, we introduce the notion of a matching initialization with the following deﬁnitions.
Revised: In order to compare networks trained on different subsets of data for different numbersof iterations, we introduce the notion of a matching initialization with the following deﬁnitions.

Original: Deﬁnition 3.1 Let w S t be the dense network weights after training on a subset of the training data,
Revised: Deﬁnition 3.1 Let w St be the dense network weights after training on a subset of the training data,

Original: S until rewind step t .
Revised: S until rewind step t .

Original: Then for two data subsets { S, S 0 } , rewind times { t, t 0 } and a given range ofsparsities, w S 0 t 0 is said to dominate (weakly dominate) w St if sparse networks obtained from IMPwithw S 0 t 0 as the initialization achieve better (no-worse) accuracy than those obtained from IMP withw Stas the initialization.
Revised: Then for two data subsets { S, S 0 } , rewind times { t, t 0 } and a given range ofsparsities, w S 0 t 0 is said to dominate (weakly dominate) w St if sparse networks obtained from IMP with

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: w S 0 t 0 as the initialization achieve better (no-worse) accuracy than those obtained from IMP with w S tas the initialization.

Original: For a network trained on the full dataset for t steps, we write w t .
Revised: For a network trained on the full dataset for t steps, we write w t .

Original: In Figure 2, we see that w t ⇤dominates w t ⇤ / 2 which in turn dominates w 0 .
Revised: In Figure 2, we see that w t ⇤dominates w t ⇤ / 2 which in turn dominates w 0 .

Original: We investigate which data subsets S and rewindsteps t lead to networks w St that dominate w t ⇤ and w t ⇤ / 2 —such networks are called matchinginitializations.
Revised: We investigate which data subsets S and rewind steps tlead to networks w St that dominate w t ⇤ and w t ⇤ / 2 —such networks are called matching initializations.

Original: [Figure]
Revised: [Figure]

Original: Deﬁnition 3.2 A dense network w St is a matching initialization for rewind time
Revised: Deﬁnition 3.2 A dense network w St is a matching initialization for rewind time

Original: t
Revised: t

Original: ⇤ if w St weaklydominates w t ⇤ .
Revised: ⇤ if w St weaklydominates w t ⇤ .

Original: We empirically ﬁnd that certain surprisingly small subsets S and rewind step t r < t ⇤ lead to matchinginitializations for rewind time t ⇤ .
Revised: We empirically ﬁnd that certain surprisingly small subsets S and rewind step t r < t ⇤ lead tomatchining initializations fore rewind time t ⇤ .

Original: Experimental design.
Revised: Experimental design.

Original: To evaluate the effect of the training subset size and composition on thequality of the pre-trained initialization, we train ResNet-20/ResNet-32/ResNet-56 on subsets of
Revised: To evaluate the effect of the training subset size and composition on thequality of the pre-trained initialization, we train ResNet-20/ResNet-32/ResNet-56 on subsets of

Original: CIFAR-10/CIFAR-100/CINIC-10, respectively.
Revised: CIFAR-10/CIFAR-100/CINIC-10, respectively.

Original: The subset size M is varied and subsets are chosenas follows: (i) M randomly selected examples, distributed equally among all classes; (ii) the easiest
Revised: The subset size M is varied and subsets are chosenas follows: (i) M randomly selected examples, distributed equally among all classes; (ii) the easiest

Original: M examples; (iii) the hardest M examples.
Revised: M examples; (iii) the hardest M examples.

Original: The easiest examples are those with the smallest EL2Nscores and the hardest are the examples with the largest EL2N scores [23].
Revised: The easiest examples are those with the smallest EL2Nscores and the hardest are the examples with the largest EL2N scores [14].

Original: Due to the signiﬁcant computational demands of performing IMP with multiple pre-training schemesand replicates, we focus on a targeted set of pre-training iterations t r .
Revised: Due to the signiﬁcant computational demands of performing IMP with multiple pre-training schemesand replicates, we focus on a targeted set of pre-training iterations t r .

Original: In particular, we study thepre-training iteration t r = t
Revised: In particular, we study thepre-training iteration t r = t

Original: ⇤ where training on all examples leads IMP to ﬁnd spare sub-networksthat perform as well as the dense network for a large range of sparsities ( t ⇤ = 400 for CIFAR-10 and800 for CIFAR-100 and CINIC-10).
Revised: ⇤ where training on all examples leads IMP to ﬁnd spare sub-networksthat perform as well as the dense network for a large range of sparsities ( t ⇤ = 400 for CIFAR-10 and800 for CIFAR-100 and CINIC-10).

Original: We also study the more challenging pre-training iteration oft r = t ⇤ 2 , where pre-training on all data does not yield a matching initialization.
Revised: We also study the more challenging pre-training iteration oft r = t ⇤ 2 , where pre-training on all data does not yield a matching initialization.

Original: When M examplesare not enough to train for t r iterations without replacement, we make multiple passes over the Mexamples as necessary.
Revised: When M examplesare not enough to train for t r iterations without replacement, we make multiple passes over the Mexamples as necessary.

Original: Figure 2 shows the best performing easy data subset for each dataset acrossthe full range of sparsities; Figure 3 shows the performance across subset size at three ﬁxed sparsities.
Revised: Figure 2 shows the best performing easy data subset for each dataset acrossthe full range of sparsities; Figure 3 shows the performance across subset size at three ﬁxed sparsities.

Original: Randomly chosen examples.
Revised: Randomly chosen examples.

Original: Pre-training the dense networks on small, randomly chosen subsets Scan lead to initializations for IMP, w St r , that dominate initializations w t r , trained on the entire trainingset for the same number of steps.
Revised: Pre-training the dense networks on small, randomly chosen subsets Scan lead to initializations for IMP, w St r , that dominate initializations w t r , trained on the entire trainingset for the same number of steps.

Original: In Figure 2 we see that for all dataset + network combinations,pre-training the dense network on a small random subset (solid green curve with circles; sizes rangingfrom 4.1% for CIFAR-100 to 28.4% for CINIC-10) for t r = t ⇤ / 2 leads to initializations that (weakly)dominate those that were obtained from training the network for the same number of steps on allthe data.
Revised: In Figure 2 we see that for all dataset + network combinations,pre-training the dense network on a small random subset (solid green curve with circles; sizes rangingfrom 4.1% for CIFAR-100 to 28.4% for CINIC-10) for t r = t ⇤ / 2 leads to initializations that (weakly)dominate those that were obtained from training the network for the same number of steps on allthe data.

Original: This observation leads to a surprising suggestion: in these experiments, the subset size issmaller than the total number of images seen during the pre-training phase; for the particular goal ofﬁnding a matching initialization of IMP, multiple passes through the same small dataset can be morebeneﬁcial than seeing more random data.
Revised: This observation leads to a surprising suggestion: in these experiments, the subset size issmaller than the total number of images seen during the pre-training phase; for the particular goal ofﬁnding a matching initialization of IMP, multiple passes through the same small dataset can be morebeneﬁcial than seeing more random data.

Original: Easiest examples (lowest EL2N scores).
Revised: Easiest examples (lowest EL2N scores).

Original: By pre-training on just the easiest examples (identiﬁed bylowest EL2N scores, solid red curve with triangles in Figure 2), we can obtain matching initializationsin fewer steps compared to training on the full dataset.
Revised: By pre-training on just the easiest examples (identiﬁed bylowest EL2N scores, solid red curve with triangles in Figure 2), we can obtain matching initializationsin fewer steps compared to training on the full dataset.

Original: In Figure 2, we see that for all three dataset[Table] and network combinations and for the subset sizes shown, the initialization obtained from training onthe easiest examples for t r = t ⇤ / 2 steps leads to matching initializations for t ⇤ .
Revised: In Figure 2, we see that for all three dataset and network combinations and for the subset sizes shown, the initialization obtained from training onthe easiest examples for t r = t ⇤ / 2 steps leads to matching initializations for t ⇤ .

Original: Hardest examples (highest EL2N scores).
Revised: Hardest examples (highest EL2N scores).

Original: Conversely, pre-training on the hardest examples (solidpink curve with crosses in Figure 2) yields worse accuracies than pre-training on all examples ora random subset.
Revised: Conversely, pre-training on the hardest examples (solidpink curve with crosses in Figure 2) yields worse accuracies than pre-training on all examples ora random subset.

Original: In fact, on CIFAR-10 the hardest examples perform little better than using nopre-training at all.
Revised: In fact, on CIFAR-10 the hardest examples perform little better than using nopre-training at all.

Original: Interestingly, when training a dense network, these hardest examples are crucialfor obtaining a network with good generalization properties [23].
Revised: Interestingly, when training a dense network, these hardest examples are crucialfor obtaining a network with good generalization properties [14].

Original: This suggests that while the hardexample may be key later in training, repeated passes through easier examples should be the focusduring the very early stages of training to quickly ﬁnd a good initialization for IMP.
Revised: This suggests that while the hardexample may be key later in training, repeated passes through easier examples should be the focusduring the very early stages of training to quickly ﬁnd a good initialization for IMP.

Original: Randomized labels during pre-training.
Revised: Randomized labels during pre-training.

Original: Pre-training on a all data or a random subset is not robustto corruption with random label noise [28] during the pre-training phase.
Revised: Pre-training on a all data or a random subset is not robustto corruption with random label noise [17] during the pre-training phase.

Original: As seen in Figure 4, thehigher the percentage of randomized labels, the lower the performance of these data subsets, andin particular, the pre-trained rewinding point becomes no better than a random initialization when50% of the labels are randomized.
Revised: As seen in Figure 4, thehigher the percentage of randomized labels, the lower the performance of these data subsets, andin particular, the pre-trained rewinding point becomes no better than a random initialization when50% of the labels are randomized.

Original: On the other hand, training on easiest data with EL2N scorescomputed on the corrupted dataset is robust to this noise (solid red curve with triangles in Figure 4).
Revised: On the other hand, training on easiest data with EL2N scorescomputed on the corrupted dataset is robust to this noise (solid red curve with triangles in Figure 4).

Original: This is because examples with randomized labels are hard ([23]) according to this metric, and thusthe easiest examples will select a subset of largely uncorrupted data.
Revised: This is because examples with randomized labels are hard ([14]) according to this metric, and thusthe easiest examples will select a subset of largely uncorrupted data.

Original: Summary.
Revised: Summary.

Original: Taken together, our results suggest that, ﬁnding a matching initialization for IMP atrewinding step t ⇤ is an interesting problem in which “more data, more training” is not optimal; it isneither necessary to train on all the data nor to train for the full t ⇤ steps.
Revised: Taken together, our results suggest that, ﬁnding a matching initialization for IMP atrewinding step t ⇤ is an interesting problem in which “more data, more training” is not optimal; it isneither necessary to train on all the data nor to train for the full t ⇤ steps.

Original: In fact, we can get away withtraining on a surprisingly small dataset for as little as half the number of steps if we make multiplepasses through the right examples, here the easiest examples as deﬁned by  lowest EL2N scores.
Revised: In fact, we can get away withtraining on a surprisingly small dataset for as little as half the number of steps if we make multiplepasses through the right examples, in this case the easiest examples as deﬁned by the lowest EL2Nscores.

Revised: In Appendix B.1, we show that pre-training on easy data increases the norm of the minibatchgradient, refuting the hypothesis that pre-training on easy data effectively performs gradient clipping.

Revised: See Appendix B for additional results on hypotheses for the role of easy data.

Original:  Pre-training through the lens of Linear Mode Connectivity
Revised:  Pre-training through the lens of Linear Mode Connectivity

Original: Frankle et al.
Revised: Frankle et al.

Original: [10] observed a strong indicator of whether a sparse sub-network generated by IMPwould be able to match the accuracy of the dense network is whether or not that sparse sub-networktrain to the same linearly connected mode (i.e. the train loss barrier is close to 0, see Figure 1).
Revised: [6] observed a strong indicator of whether a sparse sub-network generated by IMPwould be able to match the accuracy of the dense network is whether or not that sparse sub-networktrain to the same linearly connected mode (i.e. the train loss barrier is close to 0, see Figure 1).

Original: Herewe ask what properties of the dense network at the rewind point might also be predictive of thisproperty.
Revised: Herewe ask what properties of the dense network at the rewind point might also be predictive of thisproperty.

Original: In the architectures + datasets we consider, the onset of linear mode connectivity (LMC)occurs in the dense network later than the ﬁrst rewind time which produces a good initializationfor IMP (i.e. ¯ t > t r ), and thus, we consider two generalizations to the notion of LMC.
Revised: In the architectures + datasets we consider, the onset of linear mode connectivity (LMC)occurs in the dense network later than the ﬁrst rewind time which produces a good initializationfor IMP (i.e. ¯ t > t r ), and thus, we consider two generalizations to the notion of LMC.

Original: First, welook at the train loss barrier on a per-example basis meaning that instead of considering the full losslandscape we separately look at the performance of the linear interpolation on each example (theoriginal notion of train loss barrier is obtained by averaging these values).
Revised: First, welook at the train loss barrier on a per-example basis meaning that instead of considering the full losslandscape we separately look at the performance of the linear interpolation on each example (theoriginal notion of train loss barrier is obtained by averaging these values).

Original: Second, we then look atthe distribution of these per-example train loss barriers as a continuous measure of the state of thenetwork rather than simply considering whether their average is 0.
Revised: Second, we then look atthe distribution of these per-example train loss barriers as a continuous measure of the state of thenetwork rather than simply considering whether their average is 0.

Original: We demonstrate in general thatthese measures of the loss landscape of the dense network correlate well with the IMP performanceof the pre-trained initialization.
Revised: We demonstrate in general thatthese measures of the loss landscape of the dense network correlate well with the IMP performanceof the pre-trained initialization.

Original: We additionally show that pre-training on easy data results in adistribution with a large fraction of examples linearly mode connected faster than pre-training on alldata.

Original: See Appendices B and C for additional experiments and an exploration of why this occurs.

Original: The distribution of per-example loss barriers.
Revised: The distribution of per-example loss barriers.

Original: In the left and center columns of Figure 5 we plothistograms of per-example loss barriers for various rewinding points and data subsets for CIFAR-100(panel a) and CINIC-10 (panel b).
Revised: In the left and center columns of Figure 5 we plothistograms of per-example loss barriers for various rewinding points and data subsets for CIFAR-100(panel a) and CINIC-10 (panel b).

Original: There is a clear shift in the empirical distribution of the per-exampleloss barriers — not only the mean is shifting as previously observed, but also the mode.
Revised: There is a clear shift in the empirical distribution of the per-exampleloss barriers — not only the mean is shifting as previously observed, but also the mode.

Original: Pre-trainingon all data for t ⇤ iterations has the same effect on the distribution of the loss barriers as pre-trainingon the easy examples for t ⇤ / 2 iterations, two procedures which produce matching IMP initializations.
Revised: Pre-trainingon all data for t ⇤ iterations has the same effect on the distribution of the loss barriers as pre-trainingon the easy examples for t ⇤ / 2 iterations, two procedures which produce matching IMP initializations.

Original: Given the connection between the error barrier and IMP sub-network performance, it is natural toask whether examples which achieve low loss barrier early in training with the full dataset are moreimportant for producing a matching initialization than high loss barrier examples.
Revised: Given the connection between the error barrier and IMP sub-network performance, it is natural toask whether examples which achieve low loss barrier early in training with the full dataset are moreimportant for producing a matching initialization than high loss barrier examples.

Original: We call this rankingof the data by the per-example loss barrier at a given iteration the LMC score, and the blue squaresin the scatter plot are the result of training on a subset of the data determined by this score.
Revised: We call this rankingof the data by the per-example loss barrier at a given iteration the LMC score, and the blue squaresin the scatter plot are the result of training on a subset of the data determined by this score.

Original: Ourempirical ﬁndings suggest that using this score does not match the performance of using easy data(see Appendix D).
Revised: Ourempirical ﬁndings suggest that using this score does not match the performance of using easy data(see Appendix C).

Original: The correlation between ﬁnal test accuracy and mean train loss barrier.
Revised: The correlation between ﬁnal test accuracy and mean train loss barrier.

Original: The right column of
Revised: The right column of

Original: Figure 5 shows a scatter plot of ﬁnal test accuracy of the sparse trained network vs. two differentproperties of the dense initialization: test accuracy of this pre-train point (top) and the average train
Revised: Figure 5 shows a scatter plot of ﬁnal test accuracy of the sparse trained network vs. two differentproperties of the dense initialization: test accuracy of this pre-train point (top) and the average train

Original: [Figure]
Revised: [Figure]

Original: steps leads to a distribution shift to smaller per-examples train loss barriers between children runs of the dense network.
Revised: steps leads to a distribution shift to smaller per-examples train loss barriers between children runs of the dense network.

Original: When pre-trained for just t ⇤ / 2 iterations but with a random subset of data, the distribution is similar to pre-training on all data for t ⇤ / 2 steps.
Revised: When pre-trained for just t ⇤ / 2 iterations but with a random subset of data, the distribution is similar to pre-training on all data for t ⇤ / 2 steps.

Original: However, when pre-trained on a subset of easy examples for t ⇤ / 2 iterations, the distribution of train loss barriers displays a shift comparable to pre-training on all the data for t ⇤ iterations.
Revised: However, when pre-trained on a subset of easy examples for t ⇤ / 2 iterations, the distribution of train loss barriers displays a shift comparable to pre-training on all the data for t ⇤ iterations.

Original: As seen in Figure 2 and the Appendix, pre-training with this random subset matches pre
Revised: As seen in Figure 2 and the Appendix, pre-training with this random subset matches pre

Original: -training with all data for t ⇤ / 2 steps while pre-training with the easiest data for t ⇤ / 2 matches pre-training on all data for t ⇤ steps.
Revised: -training with all data for t ⇤ / 2 steps while pre-training with the easiest data for t ⇤ / 2 matches pre-training on all data for t ⇤ steps.

Original: This observation suggests that the per-example distribution of train loss barriers may be a useful signature for predicting the 
Revised: This observation suggests that the per-example distribution of train loss barriers may be a useful signature for predicting the train accuracy

Original: IMP performance

Original: We observe that the mean train loss barrier is better correlated with the ﬁnal test accuracy than the pre-train accuracy of the dense initialization.
Revised: We observe that the mean train loss barrier is better correlated with the ﬁnal test accuracy than the pre-train accuracy of the dense initialization.

Original: loss barrier from 3 pairs of children runs (bottom).
Revised: loss barrier from 3 pairs of children runs (bottom).

Original: Here we observe the same result across thedifferent training conditions: a correlation between the train loss barrier when spawning from w t r ,and the ﬁnal test accuracy of the sparse sub-networks.
Revised: Here we observe the same result across thedifferent training conditions: a correlation between the train loss barrier when spawning from w t r ,and the ﬁnal test accuracy of the sparse sub-networks.

Original: The correlation is weaker for the pre-train testaccuracy, suggesting that the loss landscape properties hold more import for determining the successof IMP.
Revised: The correlation is weaker for the pre-train testaccuracy, suggesting that the loss landscape properties hold more import for determining the successof IMP.

Original: Furthermore, we see that training on the easiest examples produces rewinding points withsmaller train loss barriers than training on random subsets.
Revised: Furthermore, we see that training on the easiest examples produces rewinding points withsmaller train loss barriers than training on random subsets.

Original:  The Role of Training Data in Learning Rate Warmup
Revised:  The Role of Training Data in Learning Rate Warmup

Original: To this point, we have studied the early phase of training through the lens of the lottery tickethypothesis.
Revised: To this point, we have studied the early phase of training through the lens of the lottery tickethypothesis.

Original: However, the dynamics of the early phase of training have important implications beyondthe sparse regime considered by lottery ticket research.
Revised: However, the dynamics of the early phase of training have important implications beyondthe sparse regime considered by lottery ticket research.

Original: For example, learning rate warmup is a nearlyubiquitous part of training state-of-the-art models.
Revised: For example, learning rate warmup is a nearlyubiquitous part of training state-of-the-art models.

Original: Warmup can be viewed as an accommodation forthe idiosyncrasies of the earliest part of training and a pre-training strategy that prepares the networkto train at the full learning rate.
Revised: Warmup can be viewed as an accommodation forthe idiosyncrasies of the earliest part of training and a pre-training strategy that prepares the networkto train at the full learning rate.

Original: In this section, we extend our previous experiments to dense trainingwith learning rate warmup by ablating the training data during this phase to assess whether easy dataalone sufﬁces, similar to our observations in the pre-training phase for IMP (Section 3).
Revised: In this section, we extend our previous experiments to dense trainingwith learning rate warmup by ablating the training data during this phase to assess whether easy dataalone sufﬁces, similar to our observations in the pre-training phase for IMP (Section 3).

Original: Easy data does not reduce the amount of training required to stabilize the network via learningrate warmup.
Revised: Easy data does not reduce the amount of training required to stabilize the network via learningrate warmup.

Original: The effects of performing learning rate warmup with easy, random, and hardestsubsets of the data appear in Figure 6.
Revised: The effects of performing learning rate warmup with easy, random, and hardestsubsets of the data appear in Figure 6.

Original: Intriguingly, the most striking performance drop is observedwhen doing learning rate warmup with easy data, especially at longer pre-training times.
Revised: Intriguingly, the most striking performance drop is observedwhen doing learning rate warmup with easy data, especially at longer pre-training times.

Original: Pre-trainingon the random and hardest data produces the same performance for sufﬁciently large subset sizes.
Revised: Pre-trainingon the random and hardest data produces the same performance for sufﬁciently large subset sizes.

Original: The role played by data during IMP pre-training is thus different than that during learning ratewarmup; the easy data does not add stability beneﬁts during the later.
Revised: The role played by data during IMP pre-training is thus different than that during learning ratewarmup; the easy data does not add stability beneﬁts during the later.

Original: We hypothesize that this isbecause learning rate warmup happens over a much larger fraction of the total training time than IMPpre-training.
Revised: We hypothesize that this isbecause learning rate warmup happens over a much larger fraction of the total training time than IMPpre-training.

Original: See Appendix E for extended discussion.

Original: [Figure]
Revised: [Figure]

Original: a much longer fraction of training than IMP pre-training which lasts for only a few hundred steps.
Revised: IMP pre-training which lasts for only a few hundred steps.

Original:  Discussion
Revised:  Discussion

Original: Recent empirical evidence has shown that deep neural network optimization proceeds in severaldistinct phases of training [8, 11].
Revised: Recent empirical evidence has shown that deep neural network optimization proceeds in severaldistinct phases of training [4, 7].

Original: Understanding the role that data plays in these different phasescan help us characterize what is being learned during them.
Revised: Understanding the role that data plays in these different phasescan help us characterize what is being learned during them.

Original: Since data loading is often a bottleneck,this understanding also has the potential enable more efﬁcient training schemes.
Revised: Since data loading is often a bottleneck,this understanding also has the potential enable more efﬁcient training schemes.

Original: In this work, wehave considered two essential phases of early training: pre-training for IMP which enables sparseoptimization and learning rate warmup which stabilizes network training.
Revised: In this work, wehave considered two essential phases of early training: pre-training for IMP which enables sparseoptimization and learning rate warmup which stabilizes network training.

Original: Our experiments identifywhat data is sufﬁcient for each of these procedures, and as they both occur early in training, theseﬁnding can be used to design more efﬁcient data loaders for streaming datasets.
Revised: Our experiments identifywhat data is sufﬁcient for each of these procedures, and as they both occur early in training, theseﬁnding can be used to design more efﬁcient data loaders for streaming datasets.

Original: Furthermore, as
Revised: Furthermore, as

Original: IMP is a computationally expensive procedure, understanding how it works is essential for designingbetter algorithms with the same performance.
Revised: IMP is a computationally expensive procedure, understanding how it works is essential for designingbetter algorithms with the same performance.

Original: To this end, we identify loss landscape properties ofthe dense network initialization for IMP that are predictive of successful spare training.
Revised: To this end, we identify loss landscape properties ofthe dense network initialization for IMP that are predictive of successful spare training.

Original: Though thiswork does not provide an improved algorithm for obtaining sparse networks, we believe our resultsprovide essential guidance for researchers pursuing algorithms that perform pruning early in training(i.e.
Revised: Though thiswork does not provide an improved algorithm for obtaining sparse networks, we believe our resultsprovide essential guidance for researchers pursuing algorithms that perform pruning early in training(i.e.

Original: ﬁnding sparse masks without training to convergence).
Revised: ﬁnding sparse masks without training to convergence).