Original: Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks
Revised: Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks

Original: Abstract
Revised: Abstract

Original: A striking observation about iterative magnitude pruning (IMP; Frankle et al. 10) is that—after just a few hundred steps of dense training—the method can find a sparse sub-network that can be trained to the same accuracy as the dense network.
Revised: A striking observation about iterative magnitude pruning (IMP; Frankle et al. 10) is that—after just a few hundred steps of dense training—the method can ﬁnd a sparse sub-network that can be trained to the same accuracy as the dense network.

Original: However, the same does not hold at step 0, i.e., random initialization.
Revised: However, the same does not hold at step 0, i.e., random initialization.

Original: In this work, we seek to understand how this early phase of pre-training leads to a good initialization for IMP both through the lens of the data distribution and the loss landscape geometry.
Revised: In this work, we seek to understand how this early phase of pre-training leads to a good initialization for IMP both through the lens of the data distribution and the loss landscape geometry.

Original: Empirically we observe that, holding the number of pretraining iterations constant, training on a small fraction of (randomly chosen) data suffices to obtain an equally good initialization for IMP.
Revised: Empirically we observe that, holding the number of pretraining iterations constant, training on a small fraction of (randomly chosen) data sufﬁces to obtain an equally good initialization for IMP.

Original: We additionally observe that by pre-training only on “easy” training data we can decrease the number of steps necessary to find a good initialization for IMP compared to training on the full dataset or a randomly chosen subset.
Revised: We additionally observe that by pre-training only on “easy” training data we can decrease the number of steps necessary to ﬁnd a good initialization for IMP compared to training on the full dataset or a randomly chosen subset.

Original: Finally, we identify novel properties of the loss landscape of dense networks that are predictive of IMP performance, showing in particular that more examples being linearly mode connected in the dense network correlates well with good initializations for IMP.
Revised: Finally, we identify novel properties of the loss landscape of dense networks that are predictive of IMP performance, showing in particular that more examples being linearly mode connected in the dense network correlates well with good initializations for IMP.

Original: Combined, these results provide new insight into the role played by the early phase training in IMP.
Revised: Combined, these results provide new insight into the role played by the early phase training in IMP.

Original:  Introduction
Revised:  Introduction

Original: Modern deep neural networks are often trained in the massively over-parameterized regime.
Revised: Modern deep neural networks are often trained in the massively over-parameterized regime.

Original: Though these networks can eventually be pruned, quantized, or distilled into smaller networks, the resources required for the initial over-parameterized training poses a challenge to the democratization and sustainability of AI.
Revised: Though these networks can eventually be pruned, quantized, or distilled into smaller networks, the resources required for the initial over-parameterized training poses a challenge to the democratization and sustainability of AI.

Original: This raises a fundamental question: under what circumstances can we efficiently train sparse networks?
Revised: This raises a fundamental question: under what circumstances can we efﬁciently train sparse networks?

Original: Recent work on the lottery ticket hypothesis [9, 10] has shown that, after just a few hundred steps of pre-training, a dense network contains a sparse sub-network that can be trained without any loss in performance.
Revised: Recent work on the lottery ticket hypothesis [9, 10] has shown that, after just a few hundred steps of pre-training, a dense network contains a sparse sub-network that can be trained without any loss in performance.

Original: Finding this sparse sub-network currently requires multiple rounds of training to convergence, pruning, and rewinding to the pre-train point, a procedure termed iterative magnitude pruning (IMP, Figure 1; [9, 10]), Remarkably, even after all these rounds of training, we do not find trainable sparse sub-networks if we rewind to the random initialization; the first few hundred steps of dense network training is essential for finding sparse networks through IMP.
Revised: Finding this sparse sub-network currently requires multiple rounds of training to convergence, pruning, and rewinding to the pre-train point, a procedure termed iterative magnitude pruning (IMP, Figure 1; [9, 10]), Remarkably, even after all these rounds of training, we do not ﬁnd trainable sparse sub-networks if we rewind to the random initialization; the ﬁrst few hundred steps of dense network training is essential for ﬁnding sparse networks through IMP.

Original: In this work, we seek to understand this very short but critical phase of pre-training.
Revised: In this work, we seek to understand this very short but critical phase of pre-training.

Original: In particular, we investigate the effect of training data and number of steps used during pre-training on the accuracy achieved by IMP.
Revised: In particular, we investigate the effect of training data and number of steps used during pre-training on the accuracy achieved by IMP.

Original: We also explain how certain properties of the loss landscape allow us to predict whether we will find a matching initialization (i.e., an initialization from which IMP “succeeds” in finding a subnetwork that can match the accuracy of the unpruned network, to be formalized later).
Revised: We also explain how certain properties of the loss landscape allow us to predict whether we will ﬁnd a matching initialization (i.e., an initialization from which IMP “succeeds” in ﬁnding a subnetwork that can match the accuracy of the unpruned network, to be formalized later).

Original: IMP proceeds in three phases: an initial pre-training phase where (1) the dense weights are trained for a few hundred steps, followed by (2) the resource-intensive mask search phase , during which we
Revised: IMP proceeds in three phases: an initial pre-training phase where (1) the dense weights are trained for a few hundred steps, followed by (2) the resource-intensive mask search phase , during which we

Original: Iteration k
Revised: Iteration k

Original: Iteration k
Revised: Iteration k

Original: Iteration T
Revised: Iteration T

Original: P r un e a n d R e w i n d
Revised: P r un e a n d R e w i n d

Original: Iteration
Revised: Iteration

Original: Iteration T
Revised: Iteration T

Original: Iteration k
Revised: Iteration k

Original: Iteration T
Revised: Iteration T

Original: Pruning Round
Revised: Pruning Round

Original: Final Mask
Revised: Final Mask

Original: P r un e a n d R e w i n d
Revised: P r un e a n d R e w i n d

Original: Phase 1: Pre-training
Revised: Phase 1: Pre-training

Original: Phase 3:
Revised: Phase 3:

Original: Sparse Training
Revised: Sparse Training

Original: P h a s e 2 :
Revised: P h a s e 2 :

Original: M a s k S e a r c h
Revised: M a s k S e a r c h

Original: Train Loss Barrier iteratively find a sparse mask by training the network and rewinding the weights repeatedly, finally, ending with (3) the sparse training phase , when the sparse masked network is trained.
Revised: Train Loss Barrier iteratively ﬁnd a sparse mask by training the network and rewinding the weights repeatedly, ﬁnally, ending with (3) the sparse training phase , when the sparse masked network is trained.

Original: The weights learned after  pre-training phase—which serve as the initialization for  mask search phase—are referred to as the pre-trained initialization (see Figure 1).
Revised: The weights learned after the pre-training phase—which serve as the initialization for the mask search phase—are referred to as the pre-trained initialization (see Figure 1).

Original: In this work, we focus on studying the pre-training phase.
Revised: In this work, we focus on studying the pre-training phase.

Original: To probe sufficient information needed to arrive at a matching initialization for IMP, we modify the pre-training phase by training with different pruned datasets and varying the number of pre-training steps; we then compare the performance of the resulting sub-networks across a range of sparsities.
Revised: To probe sufﬁcient information needed to arrive at a matching initialization for IMP, we modify the pre-training phase by training with different pruned datasets and varying the number of pre-training steps; we then compare the performance of the resulting sub-networks across a range of sparsities.

Original: Following Paul et al.
Revised: Following Paul et al.

Original: [23], we prune the datasets both randomly and according to EL2N scores, which estimate example difficulty by measuring early training performance in an ensemble of networks.
Revised: [23], we prune the datasets both randomly and according to EL2N scores, which estimate example difﬁculty by measuring early training performance in an ensemble of networks.

Original: We next turn to characterizing the loss landscape properties of matching initialization for IMP.
Revised: We next turn to characterizing the loss landscape properties of matching initialization for IMP.

Original: Frankle et al.
Revised: Frankle et al.

Original: [10] investigate the relationship between successful initializations for IMP and linear mode connectivity on the sparse initialization for the entire dataset; here, we investigate this relationship on the dense initialization on a per-example basis.
Revised: [10] investigate the relationship between successful initializations for IMP and linear mode connectivity on the sparse initialization for the entire dataset; here, we investigate this relationship on the dense initialization on a per-example basis.

Original: This enables us to identify signatures of the dense loss landscape correlated with a matching initialization.
Revised: This enables us to identify signatures of the dense loss landscape correlated with a matching initialization.

Original: Thinking more broadly, we explore the relationship between sparse trainability and training stability.
Revised: Thinking more broadly, we explore the relationship between sparse trainability and training stability.

Original: Gilmer et al.
Revised: Gilmer et al.

Original: [13] demonstrate that, in hyperparameter regimes in which early training is unstable, learning rate warmup helps stabilize training.
Revised: [13] demonstrate that, in hyperparameter regimes in which early training is unstable, learning rate warmup helps stabilize training.

Original: We build on this by investigating if the same pruned datasets which decreased the amount of pre-training required to find a matching initialization for IMP also decrease the required duration of learning rate warmup.
Revised: We build on this by investigating if the same pruned datasets which decreased the amount of pre-training required to ﬁnd a matching initialization for IMP also decrease the required duration of learning rate warmup.

Original: Overall, this empirical analysis provides new insight into the important and mysterious role that pre-training the dense network plays in IMP.
Revised: Overall, this empirical analysis provides new insight into the important and mysterious role that pre-training the dense network plays in IMP.

Original: Intriguingly, finding a matching initialization for IMP is an example of a problem in which the dominant paradigm of “more data, more training” is not optimal.
Revised: Intriguingly, ﬁnding a matching initialization for IMP is an example of a problem in which the dominant paradigm of “more data, more training” is not optimal.

Original: Indeed, by carefully choosing the training examples, we can not only use a very small subset of the data but also reduce the training time required to solve this problem.
Revised: Indeed, by carefully choosing the training examples, we can not only use a very small subset of the data but also reduce the training time required to solve this problem.

Original: From a scientific perspective, this suggests that different phases of training play different roles in the optimization process.
Revised: From a scientiﬁc perspective, this suggests that different phases of training play different roles in the optimization process.

Original: Identifying and characterizing these phases will not only lead to a deeper understanding of the optimization of deep neural networks but also allow us to design training strategies that are optimal for each phase.
Revised: Identifying and characterizing these phases will not only lead to a deeper understanding of the optimization of deep neural networks but also allow us to design training strategies that are optimal for each phase.

Original: From the practical standpoint, data loading is often an expensive part of the training process and can be a bottleneck especially early in training.
Revised: From the practical standpoint, data loading is often an expensive part of the training process and can be a bottleneck especially early in training.

Original: By identifying the essential subset for this phase of training, our work may enable strategies to eliminate this bottleneck.
Revised: By identifying the essential subset for this phase of training, our work may enable strategies to eliminate this bottleneck.

Original: Contributions.
Revised: Contributions.

Original: We find empirical evidence for the following statements:
Revised: We ﬁnd empirical evidence for the following statements:

Original: • In the pre-training phase, only a small fraction of the data is required to find a matching initialization for IMP: On standard benchmarks, across all sparsity levels we evaluated, we find that we can match accuracy by training on a small fraction of all of the available training data, selected randomly.
Revised: • In the pre-training phase, only a small fraction of the data is required to ﬁnd a matching initialization for IMP: On standard benchmarks, across all sparsity levels we evaluated, we ﬁnd that we can match accuracy by training on a small fraction of all of the available training data, selected randomly.

Original: (As we vary the amount of training data in pre-training phase, the number
Revised: (As we vary the amount of training data in pre-training phase, the number

Original: of training iterations is held fixed.)
Revised: of training iterations is held ﬁxed.)

Original: Note that this observation changes if random label noise is introduced, in which case it becomes important to select easy (small EL2N score) examples.
Revised: Note that this observation changes if random label noise is introduced, in which case it becomes important to select easy (small EL2N score) examples.

Original: • The length of the pre-training phase can be reduced if we train only on the easiest examples: Informally, training on a small subset of “easy-to-learn” training examples produces a better rewinding point than training on all data for the same number of iterations; • The quality of a pre-trained initialization for IMP correlates with more examples being linearly mode connected in the dense network: This result complements the empirical evidence produced by Frankle et al.
Revised: • The length of the pre-training phase can be reduced if we train only on the easiest examples: Informally, training on a small subset of “easy-to-learn” training examples produces a better rewinding point than training on all data for the same number of iterations; • The quality of a pre-trained initialization for IMP correlates with more examples being linearly mode connected in the dense network: This result complements the empirical evidence produced by Frankle et al.

Original: [10] connecting linear mode connectivity and the performance of IMP.
Revised: [10] connecting linear mode connectivity and the performance of IMP.

Original: • Training on easy data, which reduces the amount of pre-training required to find a matching initialization for IMP, does not reduce the amount of training required to stablize the network via learning rate warmup ([13]): The role played by data during IMP pre-training is thus different than that during learning rate warmup.
Revised: • Training on easy data, which reduces the amount of pre-training required to ﬁnd a matching initialization for IMP, does not reduce the amount of training required to stabilize the network via learning rate warmup ([13]): The role played by data during IMP pre-training is thus different than that during learning rate warmup.

Original:  Background, Methods, and Related Work
Revised:  Background, Methods, and Related Work

Original: We consider standard neural network training on image classification.
Revised: We consider standard neural network training on image classiﬁcation.

Original: Let S = { ( x n , y n ) }
Revised: Let S = { ( x n , y n ) }

Original: Nn = denote training data, let w t ∈ R D denote model parameters (weights) of the neural network, and let w 1 , w 2 , . .
Revised: Nn = denote training data, let w t 2 R D denote model parameters (weights) of the neural network, and let w 1 , w 2 , . .

Original: . be the iterates of (some variant) of SGD, minimizing the training loss , i.e., average cross-entropy loss over the training data.
Revised: . be the iterates of (some variant) of SGD, minimizing the training loss , i.e., average cross-entropy loss over the training data.

Original: For a given training example x n , let f ( w , x ) ∈ R K denote the logit outputs of the network for weights w and p ( w , x )
Revised: For a given training example x n , let f ( w , x ) 2 R K denote the logit outputs of the network for weights w and p ( w , x )

Original: = σ ( f ( w , x )) be the probability vector returned by passing the logits through the softmax operation σ .
Revised: =  ( f ( w , x )) be the probability vector returned by passing the logits through the softmax operation  .

Original: By the loss (error) landscape , we mean the training loss (error), viewed as a function of the parameters.
Revised: By the loss (error) landscape , we mean the training loss (error), viewed as a function of the parameters.

Original: By training and test error , we mean the average 0-1 classification loss.
Revised: By training and test error , we mean the average 0-1 classiﬁcation loss.

Original: Lottery ticket subnetworks.
Revised: Lottery ticket subnetworks.

Original: The lottery ticket hypothesis [9] states that any standard neural network “contains [at initialization] a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations.”
Revised: The lottery ticket hypothesis [9] states that any standard neural network “contains [at initialization] a subnetwork that is initialized such that—when trained in isolation—it can match the test accuracy of the original network after training for at most the same number of iterations.”

Original: Although such matching subnetworks (those that can train to completion on their own and reach full accuracy by following the same procedure as the unpruned network) are not known to exist in general at random initialization, they have been shown to exist after pre-training the dense network for a short amount of time (Phase 1 in Figure 1) before pruning [2, 10, 17, 25, 27].
Revised: Although such matching subnetworks (those that can train to completion on their own and reach full accuracy by following the same procedure as the unpruned network) are not known to exist in general at random initialization, they have been shown to exist after pre-training the dense network for a short amount of time (Phase 1 in Figure 1) before pruning [2, 10, 17, 25, 27].

Original: Empirical evidence for this phenomenon comes via a procedure that finds such subnetworks retroactively after training the entire network.
Revised: Empirical evidence for this phenomenon comes via a procedure that ﬁnds such subnetworks retroactively after training the entire network.

Original: This procedure, called Iterative Magnitude Pruning [IMP; 10] is based on standard iterative pruning procedures [14], and can be decomposed into three phases (Figure 1), outlined in Algorithm 1. 
Revised: This procedure, called Iterative Magnitude Pruning [IMP; 10] is based on standard iterative pruning procedures [14], and can be decomposed into three phases (Figure 1), outlined in Algorithm 1. Algorithm 1 IMP rewinding to step t r and N iterations. . Phase 3: Sparse Training

Original: This procedure reveals the accuracy of pre-training the dense network for t r iterations, pruning, and training the pruned network thereafter.
Revised: This procedure reveals the accuracy of pre-training the dense network for t r iterations, pruning, and training the pruned network thereafter.

Original: Phase 2 can be understood as an (expensive) oracle for choosing weights to prune at t r .
Revised: Phase 2 can be understood as an (expensive) oracle for choosing weights to prune at t r .

Original: Although IMP is too expensive to use as a practical way to speed up training, it provides a window into a possible minimal number of parameters and operations necessary to successfully train a network to completion in practice.
Revised: Although IMP is too expensive to use as a practical way to speed up training, it provides a window into a possible minimal number of parameters and operations necessary to successfully train a network to completion in practice.

Original: In our work, we extend this line of thinking, pursuing the minimal amount of data necessary to find and train these subnetworks.
Revised: In our work, we extend this line of thinking, pursuing the minimal amount of data necessary to ﬁnd and train these subnetworks.

Original: This is especially tantalizing due to the potential positive interactions between sparsity and minimizing the data necessary for training.
Revised: This is especially tantalizing due to the potential positive interactions between sparsity and minimizing the data necessary for training.

Original: The result is a deeper inquiry into the minimal recipe for successful training and, thereby, into the fundamental nature of neural network learning in practice.
Revised: The result is a deeper inquiry into the minimal recipe for successful training and, thereby, into the fundamental nature of neural network learning in practice.

Original: In this respect, the closest work to ours is an experiment in a larger compendium by Frankle et al.
Revised: In this respect, the closest work to ours is an experiment in a larger compendium by Frankle et al.

Original: [11] showing that the standard pre-training phase could be replaced by a much longer self-supervised phase.
Revised: [11] showing that the standard pre-training phase could be replaced by a much longer self-supervised phase.

Original: Like our experiments in Section 3, that work aims to study what makes a matching initialization.
Revised: Like our experiments in Section 3, that work aims to study what makes a matching initialization.

Original: Our approach and findings are substantially different, however: we reduce the number of examples rather than changing the labels, and we show that, not only are a small set of examples (starting at ≈ 2% ) sufficient for pre-training, but also that they make it possible to pre-train in fewer steps.
Revised: Our approach and ﬁndings are substantially different, however: we reduce the number of examples rather than changing the labels, and we show that, not only are a small set of examples (starting at ⇡ 2% ) sufﬁcient for pre-training, but also that they make it possible to pre-train in fewer steps.

Original: There are many other ways to obtain pruned neural networks [e.g., 7, 14, 16, 20, 29].
Revised: There are many other ways to obtain pruned neural networks [e.g., 7, 14, 16, 20, 29].

Original: The distinctive aspect of work on the lottery ticket hypothesis (and the one that makes it the right starting point for our inquiry) is that its goal is to uncover a minimal path from initialization to a trained network, regardless of the cost of doing so.
Revised: The distinctive aspect of work on the lottery ticket hypothesis (and the one that makes it the right starting point for our inquiry) is that its goal is to uncover a minimal path from initialization to a trained network, regardless of the cost of doing so.

Original: The aforementioned procedures target real-world efficiency for training and/or inference.
Revised: The aforementioned procedures target real-world efﬁciency for training and/or inference.

Original: Linear Mode Connectivity and the Loss Barrier.
Revised: Linear Mode Connectivity and the Loss Barrier.

Original: We investigate the error landscape using the parent–child methodology and instability analysis of Frankle et al.
Revised: We investigate the error landscape using the parent–child methodology and instability analysis of Frankle et al.

Original: [10] and Fort et al.
Revised: [10] and Fort et al.

Original: Writing err( w ) for the test error at the weights w , the (test) loss barrier between two networks w and w ′ is sup α ∈ [0 , 1] [err( α w + (1 − α ) w ′ )
Revised: Writing err( w ) for the test error at the weights w , the (test) loss barrier between two networks w and wis sup ↵ 2 [0 , 1] [err( ↵ w + (1  ↵ ) w 0 )

Original: − ( α err( w ) + (1 − α )err( w ′ ))] .
Revised:  ( ↵ err( w ) + (1  ↵ )err( w 0 ))] .

Original: This quantity measures the maximum increase in loss above the average loss along the linear path connecting the two networks on the loss landscape.
Revised: This quantity measures the maximum increase in loss above the average loss along the linear path connecting the two networks on the loss landscape.

Original: The loss barrier at iteration t (with parent weights w t ) is the loss barrier between the (children) weights w T and w ′ T , where w T and w ′ T are copies of w t trained to completion with the same procedure but different random seeds (minibatch order, GPU noise, data augmentation, etc.).
Revised: The loss barrier at iteration t (with parent weights w t ) is the loss barrier between the (children) weights w T and w 0 T , where w T and w 0 T are copies of w t trained to completion with the same procedure but different random seeds (minibatch order, GPU noise, data augmentation, etc.).

Original: See
Revised: See

Original: Figure 1 right for a visualization.
Revised: Figure 1 right for a visualization.

Original: Empirically, the loss barrier is achieved near α = 12 , and so we compute it this way.
Revised: Empirically, the loss barrier is achieved near ↵ = 12 , and so we compute it this way.

Original: The onset of linear mode connectivity (LMC) is defined to be the iteration ¯ t such that, for all t > ¯ t , the error barrier at t is zero.
Revised: The onset of linear mode connectivity (LMC) is deﬁned to be the iteration ¯ t such that, for all t > ¯ t , the error barrier at t is zero.

Original: We follow Draxler et al.
Revised: We follow Draxler et al.

Original: [4], Garipov et al.
Revised: [4], Garipov et al.

Original: [12], and
Revised: [12], and

Original: Frankle et al.
Revised: Frankle et al.

Original: [10] in considering the 0-1 loss barrier to be zero if it is less than 2%.
Revised: [10] in considering the 0-1 loss barrier to be zero if it is less than 2%.

Original: Similar to Paul et al.
Revised: Similar to Paul et al.

Original: [23], we also measure the error (0-1 loss) or cross-entropy loss barrier on individual training examples to explore the loss landscape corresponding to different subpopulations.
Revised: [23], we also measure the error (0-1 loss) or cross-entropy loss barrier on individual training examples to explore the loss landscape corresponding to different subpopulations.

Original: Ranking training examples.
Revised: Ranking training examples.

Original: We define “easy/hard data” as the data that is ranked low/high, respectively, by the EL2N score introduced by Paul et al.
Revised: We deﬁne “easy/hard data” as the data that is ranked low/high, respectively, by the EL2N score introduced by Paul et al.

Original: EL2N scores depend on the margin early in training, and, loosely speaking, higher average margin early in training means lower importance for generalization of the final trained model.
Revised: EL2N scores depend on the margin early in training, and, loosely speaking, higher average margin early in training means lower importance for generalization of the ﬁnal trained model.

Original: This connection to margin suggests that easy data is learned first (has higher margin early in training, maintained throughout the rest of training).
Revised: This connection to margin suggests that easy data is learned ﬁrst (has higher margin early in training, maintained throughout the rest of training).

Original: EL2N scores were derived from the size of the loss gradient, and are thus are highly correlated with the magnitude of the gradient.
Revised: EL2N scores were derived from the size of the loss gradient, and are thus are highly correlated with the magnitude of the gradient.

Original: Definition 2.1 (EL2N Score)
Revised: Deﬁnition 2.1 (EL2N Score)

Original: The EL2N score of a training sample ( x , y ) at iteration t is defined as E ∥ p ( w t , x ) − y ∥ 2 , where the expectation is taken over w t conditioned on the training data.
Revised: The EL2N score of a training sample ( x , y ) at iteration t is deﬁned as E k p ( w t , x )  y k 2 , where the expectation is taken over w t conditioned on the training data.

Original: To calculate EL2N scores for a dataset, we follow the process outlined in [23].
Revised: To calculate EL2N scores for a dataset, we follow the process outlined in [23].

Original: In particular, we do the following:
Revised: In particular, we do the following:

Original: Independently train K = 10 networks from different random initializations for t iterations.
Revised: Independently train K = 10 networks from different random initializations for t iterations.

Original: For each example and each network, we calculate the L2 norm of the error vector defined as ∥ p ( x ) − y ∥ 2 where y is the one-hot encoding of the label, and p ( x ) are the softmax outputs of the network evaluated on example x . 3.
Revised: For each example and each network, we calculate the L2 norm of the error vector deﬁned as k p ( x )  y k 2 where y is the one-hot encoding of the label, and p ( x ) are the softmax outputs of the network evaluated on example x . 3.

Original: For each example, the EL2N score is the average of the error vector L2 norm across the K networks.
Revised: For each example, the EL2N score is the average of the error vector L2 norm across the K networks.

Original: In our experiments (Section 3), we vary the data that is accessible in the pre-training phase of IMP defined above.
Revised: In our experiments (Section 3), we vary the data that is accessible in the pre-training phase of IMP deﬁned above.

Original: We either choose the data that we feed to the algorithm at random while preserving class balance, or based on the EL2N scores.
Revised: We either choose the data that we feed to the algorithm at random while preserving class balance, or based on the EL2N scores.

Original: Stabilizing early training via learning rate warmup.
Revised: Stabilizing early training via learning rate warmup.

Original: Learning rate warmup period can be seen as a form of pre-training, allowing one to eventually train at higher learning rates and larger batch sizes.
Revised: Learning rate warmup period can be seen as a form of pre-training, allowing one to eventually train at higher learning rates and larger batch sizes.

Original: Training with large batches and high learning rates is desirable in practice, as it allows for more efficient GPU utilization and may reduce the total number of updates needed to achieve the desired accuracy.
Revised: Training with large batches and high learning rates is desirable in practice, as it allows for more efﬁcient GPU utilization and may reduce the total number of updates needed to achieve the desired accuracy.

Original: Gilmer et al.
Revised: Gilmer et al.

Original: [13] empirically observe that learning rate warmup essentially improves the optimization by allowing the initial optimization trajectory to navigate to “flatter” optimization landscape, i.e., one with smaller highest loss Hessian eigenvalue.
Revised: [13] empirically observe that learning rate warmup essentially improves the optimization by allowing the initial optimization trajectory to navigate to “ﬂatter” optimization landscape, i.e., one with smaller highest loss Hessian eigenvalue.

Original:  The Role of Training Data Selection in Pre-Training
Revised:  The Role of Training Data Selection in Pre-Training

Original: As can be seen in Figure 2, when training sparse networks using IMP with rewind step t r = 0 , the final test accuracy of the sparse networks falls off rapidly with increasing sparsity.
Revised: As can be seen in Figure 2, when training sparse networks using IMP with rewind step t r = 0 , the ﬁnal test accuracy of the sparse networks falls off rapidly with increasing sparsity.

Original: However, as we increase the rewind step t r , the network performance improves across all sparsity levels and at a rewind step t ∗ , the network performs as well as or better than the dense network at high sparsities.
Revised: However, as we increase the rewind step t r , the network performance improves across all sparsity levels and at a rewind step t ⇤ , the network performs as well as or better than the dense network at high sparsities.

Original: Informally, training the dense network for t ∗ steps creates a matching initialization for IMP.
Revised: Informally, training the dense network for t ⇤ steps creates a matching initialization for IMP.

Original: But what does the network learn in these first t ∗ steps?
Revised: But what does the network learn in these ﬁrst t ⇤ steps?

Original: In this section, we take the first step towards answering this question by investigating which subsets of the training data are sufficient for finding a matching initialization.
Revised: In this section, we take the ﬁrst step towards answering this question by investigating which subsets of the training data are sufﬁcient for ﬁnding a matching initialization.

Original: In order to compare networks trained on different subsets of data for different numbers of iterations, we introduce the notion of a matching initialization with the following definitions.
Revised: In order to compare networks trained on different subsets of data for different numbers of iterations, we introduce the notion of a matching initialization with the following deﬁnitions.

Original: Definition 3.1 Let w St be the dense network weights after training on a subset of the training data,
Revised: Deﬁnition 3.1 Let w St be the dense network weights after training on a subset of the training data,

Original: S until rewind step t .
Revised: S until rewind step t .

Original: Then for two data subsets { S, S ′ } , rewind times { t, t ′ } and a given range of sparsities, w S ′ t ′ is said to dominate (weakly dominate) w St if sparse networks obtained from IMP with w S ′ t ′ as the initialization achieve better (no-worse) accuracy than those obtained from IMP with w St as the initialization.
Revised: Then for two data subsets { S, S 0 } , rewind times { t, t 0 } and a given range of sparsities, w S 0 t 0 is said to dominate (weakly dominate) w St if sparse networks obtained from IMP with w S 0 t 0 as the initialization achieve better (no-worse) accuracy than those obtained from IMP with w St as the initialization.

Original: [Figure]
Revised: [Figure]

Original: , network, and hyperparameter configurations.
Revised: the solid green with circles and dashed orange curves) across dataset, network, and hyperparameter conﬁgurations.

Original: Using just the easiest training examples during this phase produces a matching initialization for rewind point t ∗ in just t ∗ / 2 steps (compare the solid red with triangles and dashed blue curves).
Revised: Using just the easiest training examples during this phase produces a matching initialization for rewind point t ⇤ in just t ⇤ / 2 steps (compare the solid red with triangles and dashed blue curves).

Original: Pre-training on the hardest examples is detrimental to the performance of the initialialization (solid pink curve with crosses).
Revised: Pre-training on the hardest examples is detrimental to the performance of the initialialization (solid pink curve with crosses).

Original: IMP with rewinding to initialization (dashed yellow curve) and the dense model (dashed grey curve) are used as baselines.
Revised: IMP with rewinding to initialization (dashed yellow curve) and the dense model (dashed grey curve) are used as baselines.

Original: For each dataset + network configuration, we present the best performing easy data subset size.
Revised: For each dataset + network conﬁguration, we present the best performing easy data subset size.

Original: For a sweep across subset sizes, see Figure 3 and
Revised: For a sweep across subset sizes, see Figure 3 and

Original: the Appendix E.
Revised: the Appendix E.

Original: For a network trained on the full dataset for t steps, we write w t .
Revised: For a network trained on the full dataset for t steps, we write w t .

Original: In Figure 2, we see that w t ∗ dominates w t ∗ / 2 which in turn dominates w 0 .
Revised: In Figure 2, we see that w t ⇤ dominates w t ⇤ / 2 which in turn dominates w 0 .

Original: We investigate which data subsets S and rewind steps t lead to networks w S t that dominate w t ∗ and w t ∗ / 2 —such networks are called matching initializations.
Revised: We investigate which data subsets S and rewind steps t lead to networks w St that dominate w t ⇤ and w t ⇤ / 2 —such networks are called matching initializations.

Original: Definition 3.2 A dense network w St is a matching initialization for rewind time t ∗ if w St weakly dominates w t ∗ .
Revised: Deﬁnition 3.2 A dense network w St is a matching initialization for rewind time t ⇤ if w St weakly dominates w t ⇤ .

Original: We empirically find that certain surprisingly small subsets S and rewind step t r < t ∗ lead to matching initializations for rewind time t ∗ .
Revised: We empirically ﬁnd that certain surprisingly small subsets S and rewind step t r < t ⇤ lead to matching initializations for rewind time t ⇤ .

Original: Experimental design.
Revised: Experimental design.

Original: To evaluate the effect of the training subset size and composition on the quality of the pre-trained initialization, we train ResNet-20/ResNet-32/ResNet-56 on subsets of
Revised: To evaluate the effect of the training subset size and composition on the quality of the pre-trained initialization, we train ResNet-20/ResNet-32/ResNet-56 on subsets of

Original: CIFAR-10/CIFAR-100/CINIC-10, respectively.
Revised: CIFAR-10/CIFAR-100/CINIC-10, respectively.

Original: The subset size M is varied and subsets are chosen as follows: (i)
Revised: The subset size M is varied and subsets are chosen as follows: (i)

Original: M randomly selected examples, distributed equally among all classes; (ii) the easiest
Revised: M randomly selected examples, distributed equally among all classes; (ii) the easiest

Original: M examples; (iii) the hardest M examples.
Revised: M examples; (iii) the hardest M examples.

Original: The easiest examples are those with the smallest EL2N scores and the hardest are the examples with the largest EL2N scores [23].
Revised: The easiest examples are those with the smallest EL2N scores and the hardest are the examples with the largest EL2N scores [23].

Original: [Figure]
Revised: [Figure]

Original: Due to the significant computational demands of performing IMP with multiple pre-training schemes and replicates, we focus on a targeted set of pre-training iterations t r .
Revised: Due to the signiﬁcant computational demands of performing IMP with multiple pre-training schemes and replicates, we focus on a targeted set of pre-training iterations t r .

Original: In particular, we study the pre-training iteration t r = t ∗ where training on all examples leads IMP to find spare sub-networks that perform as well as the dense network for a large range of sparsities ( t ∗ = 400 for CIFAR-10 and 800 for CIFAR-100 and CINIC-10).
Revised: In particular, we study the pre-training iteration t r = t ⇤ where training on all examples leads IMP to ﬁnd spare sub-networks that perform as well as the dense network for a large range of sparsities ( t ⇤ = 400 for CIFAR-10 and 800 for CIFAR-100 and CINIC-10).

Original: We also study the more challenging pre-training iteration of t r = t ∗ 2 , where pre-training on all data does not yield a matching initialization.
Revised: We also study the more challenging pre-training iteration of t r = t ⇤ 2 , where pre-training on all data does not yield a matching initialization.

Original: When M examples are not enough to train for t r iterations without replacement, we make multiple passes over the M examples as necessary.
Revised: When M examples are not enough to train for t r iterations without replacement, we make multiple passes over the M examples as necessary.

Original: Figure 2 shows the best performing easy data subset for each dataset across the full range of sparsities; Figure 3 shows the performance across subset size at three fixed sparsities.
Revised: Figure 2 shows the best performing easy data subset for each dataset across the full range of sparsities; Figure 3 shows the performance across subset size at three ﬁxed sparsities.

Original: Randomly chosen examples.
Revised: Randomly chosen examples.

Original: Pre-training the dense networks on small, randomly chosen subsets S can lead to initializations for IMP, w St r , that dominate initializations w t r , trained on the entire training set for the same number of steps.
Revised: Pre-training the dense networks on small, randomly chosen subsets S can lead to initializations for IMP, w St r , that dominate initializations w t r , trained on the entire training set for the same number of steps.

Original: In Figure 2 we see that for all dataset + network combinations, pre-training the dense network on a small random subset (solid green curve with circles; sizes ranging from 4.1% for CIFAR-100 to 28.4% for CINIC-10) for t r = t ∗ / 2 leads to initializations that (weakly) dominate those that were obtained from training the network for the same number of steps on all the data.
Revised: In Figure 2 we see that for all dataset + network combinations, pre-training the dense network on a small random subset (solid green curve with circles; sizes ranging from 4.1% for CIFAR-100 to 28.4% for CINIC-10) for t r = t ⇤ / 2 leads to initializations that (weakly) dominate those that were obtained from training the network for the same number of steps on all the data.

Original: This observation leads to a surprising suggestion: in these experiments, the subset size is smaller than the total number of images seen during the pre-training phase; for the particular goal of finding a matching initialization of IMP, multiple passes through the same small dataset can be more beneficial than seeing more random data.
Revised: This observation leads to a surprising suggestion: in these experiments, the subset size is smaller than the total number of images seen during the pre-training phase; for the particular goal of ﬁnding a matching initialization of IMP, multiple passes through the same small dataset can be more beneﬁcial than seeing more random data.

Original: Easiest examples (lowest EL2N scores).
Revised: Easiest examples (lowest EL2N scores).

Original: By pre-training on just the easiest examples (identified by lowest EL2N scores, solid red curve with triangles in Figure 2), we can obtain matching initializations in fewer steps compared to training on the full dataset.
Revised: By pre-training on just the easiest examples (identiﬁed by lowest EL2N scores, solid red curve with triangles in Figure 2), we can obtain matching initializations in fewer steps compared to training on the full dataset.

Original: In Figure 2, we see that for all three dataset and network combinations and for the subset sizes shown, the initialization obtained from training on the easiest examples for t r = t ∗ / 2 steps leads to matching initializations for t ∗ .
Revised: In Figure 2, we see that for all three dataset and network combinations and for the subset sizes shown, the initialization obtained from training on the easiest examples for t r = t ⇤ / 2 steps leads to matching initializations for t ⇤ .

Original: [Figure]
Revised: [Figure]

Original: Hardest examples (highest EL2N scores).
Revised: Hardest examples (highest EL2N scores).

Original: Conversely, pre-training on the hardest examples (solid pink curve with crosses in Figure 2) yields worse accuracies than pre-training on all examples or a random subset.
Revised: Conversely, pre-training on the hardest examples (solid pink curve with crosses in Figure 2) yields worse accuracies than pre-training on all examples or a random subset.

Original: In fact, on CIFAR-10 the hardest examples perform little better than using no pre-training at all.
Revised: In fact, on CIFAR-10 the hardest examples perform little better than using no pre-training at all.

Original: Interestingly, when training a dense network, these hardest examples are crucial for obtaining a network with good generalization properties [23].
Revised: Interestingly, when training a dense network, these hardest examples are crucial for obtaining a network with good generalization properties [23].

Original: This suggests that while the hard example may be key later in training, repeated passes through easier examples should be the focus during the very early stages of training to quickly find a good initialization for IMP.
Revised: This suggests that while the hard example may be key later in training, repeated passes through easier examples should be the focus during the very early stages of training to quickly ﬁnd a good initialization for IMP.

Original: Randomized labels during pre-training.
Revised: Randomized labels during pre-training.

Original: Pre-training on a all data or a random subset is not robust to corruption with random label noise [28] during the pre-training phase.
Revised: Pre-training on a all data or a random subset is not robust to corruption with random label noise [28] during the pre-training phase.

Original: As seen in Figure 4, the higher the percentage of randomized labels, the lower the performance of these data subsets, and in particular, the pre-trained rewinding point becomes no better than a random initialization when 50% of the labels are randomized.
Revised: As seen in Figure 4, the higher the percentage of randomized labels, the lower the performance of these data subsets, and in particular, the pre-trained rewinding point becomes no better than a random initialization when 50% of the labels are randomized.

Original: On the other hand, training on easiest data with EL2N scores computed on the corrupted dataset is robust to this noise (solid red curve with triangles in Figure 4).
Revised: On the other hand, training on easiest data with EL2N scores computed on the corrupted dataset is robust to this noise (solid red curve with triangles in Figure 4).

Original: This is because examples with randomized labels are hard ([23]) according to this metric, and thus the easiest examples will select a subset of largely uncorrupted data.
Revised: This is because examples with randomized labels are hard ([23]) according to this metric, and thus the easiest examples will select a subset of largely uncorrupted data.

Original: Summary.
Revised: Summary.

Original: Taken together, our results suggest that, finding a matching initialization for IMP at rewinding step t ∗ is an interesting problem in which “more data, more training” is not optimal; it is neither necessary to train on all the data nor to train for the full t ∗ steps.
Revised: Taken together, our results suggest that, ﬁnding a matching initialization for IMP at rewinding step t ⇤ is an interesting problem in which “more data, more training” is not optimal; it is neither necessary to train on all the data nor to train for the full t ⇤ steps.

Original: In fact, we can get away with training on a surprisingly small dataset for as little as half the number of steps if we make multiple passes through the right examples, here the easiest examples as defined by lowest EL2N scores.
Revised: In fact, we can get away with training on a surprisingly small dataset for as little as half the number of steps if we make multiple passes through the right examples, here the easiest examples as deﬁned by lowest EL2N scores.

Original:  Pre-training through the lens of Linear Mode Connectivity
Revised:  Pre-training through the lens of Linear Mode Connectivity

Original: Frankle et al.
Revised: Frankle et al.

Original: [10] observed a strong indicator of whether a sparse sub-network generated by IMP would be able to match the accuracy of the dense network is whether or not that sparse sub-network train to the same linearly connected mode (i.e. the train loss barrier is close to 0, see Figure 1).
Revised: [10] observed a strong indicator of whether a sparse sub-network generated by IMP would be able to match the accuracy of the dense network is whether or not that sparse sub-network train to the same linearly connected mode (i.e. the train loss barrier is close to 0, see Figure 1).

Original: Here we ask what properties of the dense network at the rewind point might also be predictive of this property.
Revised: Here we ask what properties of the dense network at the rewind point might also be predictive of this property.

Original: In the architectures + datasets we consider, the onset of linear mode connectivity (LMC) occurs in the dense network later than the first rewind time which produces a good initialization for IMP (i.e. ¯ t > t r ), and thus, we consider two generalizations to the notion of LMC.
Revised: In the architectures + datasets we consider, the onset of linear mode connectivity (LMC) occurs in the dense network later than the ﬁrst rewind time which produces a good initialization for IMP (i.e. ¯ t > t r ), and thus, we consider two generalizations to the notion of LMC.

Original: First, we look at the train loss barrier on a per-example basis meaning that instead of considering the full loss landscape we separately look at the performance of the linear interpolation on each example (the original notion of train loss barrier is obtained by averaging these values).
Revised: First, we look at the train loss barrier on a per-example basis meaning that instead of considering the full loss landscape we separately look at the performance of the linear interpolation on each example (the original notion of train loss barrier is obtained by averaging these values).

Original: Second, we then look at the distribution of these per-example train loss barriers as a continuous measure of the state of the network rather than simply considering whether their average is 0.
Revised: Second, we then look at the distribution of these per-example train loss barriers as a continuous measure of the state of the network rather than simply considering whether their average is 0.

Original: We demonstrate in general that these measures of the loss landscape of the dense network correlate well with the IMP performance of the pre-trained initialization.
Revised: We demonstrate in general that these measures of the loss landscape of the dense network correlate well with the IMP performance of the pre-trained initialization.

Original: We additionally show that pre-training on easy data results in a
Revised: We additionally show that pre-training on easy data results in a

Original: [Figure]
Revised: [Figure]

Original: steps leads to a distribution shift to smaller per-examples train loss barriers between children runs of the dense network.
Revised: steps leads to a distribution shift to smaller per-examples train loss barriers between children runs of the dense network.

Original: When pre-trained for just t ∗ / 2 iterations but with a random subset of data, the distribution is similar to pre-training on all data for t ∗ / 2 steps.
Revised: When pre-trained for just t ⇤ / 2 iterations but with a random subset of data, the distribution is similar to pre-training on all data for t ⇤ / 2 steps.

Original: However, when pre-trained on a subset of easy examples for t ∗ / 2 iterations, the distribution of train loss barriers displays a shift comparable to pre-training on all the data for t ∗ iterations.
Revised: However, when pre-trained on a subset of easy examples for t ⇤ / 2 iterations, the distribution of train loss barriers displays a shift comparable to pre-training on all the data for t ⇤ iterations.

Original: As seen in Figure 2 and the Appendix, pre-training with this random subset matches pre
Revised: As seen in Figure 2 and the Appendix, pre-training with this random subset matches pre

Original: -training with all data for t ∗ / 2 steps while pre-training with the easiest data for t ∗ / 2 matches pre-training on all data for t ∗ steps.
Revised: -training with all data for t ⇤ / 2 steps while pre-training with the easiest data for t ⇤ / 2 matches pre-training on all data for t ⇤ steps.

Original: This observation suggests that the per-example distribution of train loss barriers may be a useful signature for predicting the
Revised: This observation suggests that the per-example distribution of train loss barriers may be a useful signature for predicting the

Original: IMP
Revised: IMP

Original: We observe that the mean train loss barrier is better correlated with the final test accuracy than the pre-train accuracy of the dense initialization.
Revised: We observe that the mean train loss barrier is better correlated with the ﬁnal test accuracy than the pre-train accuracy of the dense initialization.

Original: distribution with a large fraction of examples linearly mode connected faster than pre-training on all data.
Revised: distribution with a large fraction of examples linearly mode connected faster than pre-training on all data.

Original: See Appendices B and C for additional experiments and an exploration of why this occurs.
Revised: See Appendices B and C for additional experiments and an exploration of why this occurs.

Original: The distribution of per-example loss barriers.
Revised: The distribution of per-example loss barriers.

Original: In the left and center columns of Figure 5 we plot histograms of per-example loss barriers for various rewinding points and data subsets for CIFAR- (panel a) and CINIC-10 (panel b).
Revised: In the left and center columns of Figure 5 we plot histograms of per-example loss barriers for various rewinding points and data subsets for CIFAR- (panel a) and CINIC-10 (panel b).

Original: There is a clear shift in the empirical distribution of the per-example loss barriers — not only the mean is shifting as previously observed, but also the mode.
Revised: There is a clear shift in the empirical distribution of the per-example loss barriers — not only the mean is shifting as previously observed, but also the mode.

Original: Pre-training on all data for t ∗ iterations has the same effect on the distribution of the loss barriers as pre-training on the easy examples for t ∗ / 2 iterations, two procedures which produce matching IMP initializations.
Revised: Pre-training on all data for t ⇤ iterations has the same effect on the distribution of the loss barriers as pre-training on the easy examples for t ⇤ / 2 iterations, two procedures which produce matching IMP initializations.

Original: Given the connection between the error barrier and IMP sub-network performance, it is natural to ask whether examples which achieve low loss barrier early in training with the full dataset are more important for producing a matching initialization than high loss barrier examples.
Revised: Given the connection between the error barrier and IMP sub-network performance, it is natural to ask whether examples which achieve low loss barrier early in training with the full dataset are more important for producing a matching initialization than high loss barrier examples.

Original: We call this ranking of the data by the per-example loss barrier at a given iteration the LMC score, and the blue squares in the scatter plot are the result of training on a subset of the data determined by this score.
Revised: We call this ranking of the data by the per-example loss barrier at a given iteration the LMC score, and the blue squares in the scatter plot are the result of training on a subset of the data determined by this score.

Original: Our empirical findings suggest that using this score does not match the performance of using easy data (see Appendix D).
Revised: Our empirical ﬁndings suggest that using this score does not match the performance of using easy data (see Appendix D).

Original: The correlation between final test accuracy and mean train loss barrier.
Revised: The correlation between ﬁnal test accuracy and mean train loss barrier.

Original: The right column of
Revised: The right column of

Original: Figure 5 shows a scatter plot of final test accuracy of the sparse trained network vs. two different properties of the dense initialization: test accuracy of this pre-train point (top) and the average train loss barrier from 3 pairs of children runs (bottom).
Revised: Figure 5 shows a scatter plot of ﬁnal test accuracy of the sparse trained network vs. two different properties of the dense initialization: test accuracy of this pre-train point (top) and the average train loss barrier from 3 pairs of children runs (bottom).

Original: Here we observe the same result across the different training conditions: a correlation between the train loss barrier when spawning from w t r , and the final test accuracy of the sparse sub-networks.
Revised: Here we observe the same result across the different training conditions: a correlation between the train loss barrier when spawning from w t r , and the ﬁnal test accuracy of the sparse sub-networks.

Original: The correlation is weaker for the pre-train test accuracy, suggesting that the loss landscape properties hold more import for determining the success of IMP.
Revised: The correlation is weaker for the pre-train test accuracy, suggesting that the loss landscape properties hold more import for determining the success of IMP.

Original: Furthermore, we see that training on the easiest examples produces rewinding points with smaller train loss barriers than training on random subsets.
Revised: Furthermore, we see that training on the easiest examples produces rewinding points with smaller train loss barriers than training on random subsets.

Original:  The Role of Training Data in Learning Rate Warmup
Revised:  The Role of Training Data in Learning Rate Warmup

Original: To this point, we have studied the early phase of training through the lens of the lottery ticket hypothesis.
Revised: To this point, we have studied the early phase of training through the lens of the lottery ticket hypothesis.

Original: However, the dynamics of the early phase of training have important implications beyond the sparse regime considered by lottery ticket research.
Revised: However, the dynamics of the early phase of training have important implications beyond the sparse regime considered by lottery ticket research.

Original: For example, learning rate warmup is a nearly ubiquitous part of training state-of-the-art models.
Revised: For example, learning rate warmup is a nearly ubiquitous part of training state-of-the-art models.

Original: Warmup can be viewed as an accommodation for the idiosyncrasies of the earliest part of training and a pre-training strategy that prepares the network to train at the full learning rate.
Revised: Warmup can be viewed as an accommodation for the idiosyncrasies of the earliest part of training and a pre-training strategy that prepares the network to train at the full learning rate.

Original: In this section, we extend our previous experiments to dense training with learning rate warmup by ablating the training data during this phase to assess whether easy data alone suffices, similar to our observations in the pre-training phase for IMP (Section 3).
Revised: In this section, we extend our previous experiments to dense training with learning rate warmup by ablating the training data during this phase to assess whether easy data alone sufﬁces, similar to our observations in the pre-training phase for IMP (Section 3).

Original: Easy data does not reduce the amount of training required to stabilize the network via learning rate warmup.
Revised: Easy data does not reduce the amount of training required to stabilize the network via learning rate warmup.

Original: The effects of performing learning rate warmup with easy, random, and hardest subsets of the data appear in Figure 6.
Revised: The effects of performing learning rate warmup with easy, random, and hardest subsets of the data appear in Figure 6.

Original: Intriguingly, the most striking performance drop is observed when doing learning rate warmup with easy data, especially at longer pre-training times.
Revised: Intriguingly, the most striking performance drop is observed when doing learning rate warmup with easy data, especially at longer pre-training times.

Original: Pre-training on the random and hardest data produces the same performance for sufficiently large subset sizes.
Revised: Pre-training on the random and hardest data produces the same performance for sufﬁciently large subset sizes.

Original: The role played by data during IMP pre-training is thus different than that during learning rate warmup; the easy data does not add stability benefits during the later.
Revised: The role played by data during IMP pre-training is thus different than that during learning rate warmup; the easy data does not add stability beneﬁts during the later.

Original: We hypothesize that this is because learning rate warmup happens over a much larger fraction of the total training time than IMP pre-training.
Revised: We hypothesize that this is because learning rate warmup happens over a much larger fraction of the total training time than IMP pre-training.

Original: [Figure]
Revised: [Figure]

Original: Discussion and algorithmic implications.
Revised: Discussion and algorithmic implications.

Original: The benefits of training on easy data—faster learning and increased stability to SGD noise—are only present in the very early phase of learning.
Revised: The beneﬁts of training on easy data—faster learning and increased stability to SGD noise—are only present in the very early phase of learning.

Original: Paul et al.
Revised: Paul et al.

Original: [23] have shown that, if we continue to train on easy data only, the network will converge to a lower final test accuracy.
Revised: [23] have shown that, if we continue to train on easy data only, the network will converge to a lower ﬁnal test accuracy.

Original: So a natural question is what other scenarios do the benefits of training on easy data extend to?
Revised: So a natural question is what other scenarios do the beneﬁts of training on easy data extend to?

Original: We began to investigate this question here by studying training at high learning rates where learning can become unstable early in training and is usually stabilized by learning rate warmup [13].
Revised: We began to investigate this question here by studying training at high learning rates where learning can become unstable early in training and is usually stabilized by learning rate warmup [13].

Original: Our results show that training only on easy examples does not help and in fact hurts performance in this case.
Revised: Our results show that training only on easy examples does not help and in fact hurts performance in this case.

Original: Two possible reasons for this behavior are (1) learning rate warmup is often used for the first 10% of training, which may be too long for easy examples to be helpful (the pre-training phase in iterative magnitude pruning where easy examples are clearly beneficial can be as little as 2% of training) and (2) the instability in this case—loss diverging to infinity—is not reduced by methods that improve stability to SGD noise.
Revised: Two possible reasons for this behavior are (1) learning rate warmup is often used for the ﬁrst 10% of training, which may be too long for easy examples to be helpful (the pre-training phase in iterative magnitude pruning where easy examples are clearly beneﬁcial can be as little as 2% of training) and (2) the instability in this case—loss diverging to inﬁnity—is not reduced by methods that improve stability to SGD noise.

Original: Despite the negative results of this experiment, we think that, when combined with our previous results, the core scientific findings in our work are interesting and broadly useful to the community.
Revised: Despite the negative results of this experiment, we think that, when combined with our previous results, the core scientiﬁc ﬁndings in our work are interesting and broadly useful to the community.

Original: In particular, some algorithmic implications are as follows: (1) While developing algorithms for training sparse networks or finding matching networks at initialization, researchers might want to incorporate methods or search for initializations that improve stability to SGD noise.
Revised: In particular, some algorithmic implications are as follows: (1) While developing algorithms for training sparse networks or ﬁnding matching networks at initialization, researchers might want to incorporate methods or search for initializations that improve stability to SGD noise.

Original: Recent work in curriculum learning found that curricula provide marginal benefits unless there are additional constraints such as a training budget [26].
Revised: Recent work in curriculum learning found that curricula provide marginal beneﬁts unless there are additional constraints such as a training budget [26].

Original: Our work helps narrow the time window in which training on easy examples is beneficial and provides another example in which, for the constrained scenario of finding matching sparse networks, a simple curriculum of training on easy data first is beneficial.
Revised: Our work helps narrow the time window in which training on easy examples is beneﬁcial and provides another example in which, for the constrained scenario of ﬁnding matching sparse networks, a simple curriculum of training on easy data ﬁrst is beneﬁcial.

Original: Thus, these findings may help in the design of better training curricula.
Revised: Thus, these ﬁndings may help in the design of better training curricula.

Original: In the training of large language models, a common problem is that loss spikes and training becomes unstable.
Revised: In the training of large language models, a common problem is that loss spikes and training becomes unstable.

Original: A hypothesis for why this phenomenon occurs is that the network is presented with a bad batch or a set of bad batches that destabilize training.
Revised: A hypothesis for why this phenomenon occurs is that the network is presented with a bad batch or a set of bad batches that destabilize training.

Original: Since this is related to instability due to SGD noise (bad batches are randomly sampled) it may be possible to improve training stability by injecting easy examples into the training process.
Revised: Since this is related to instability due to SGD noise (bad batches are randomly sampled) it may be possible to improve training stability by injecting easy examples into the training process.

Original:  Discussion
Revised:  Discussion

Original: Recent empirical evidence has shown that deep neural network optimization proceeds in several distinct phases of training [8, 11].
Revised: Recent empirical evidence has shown that deep neural network optimization proceeds in several distinct phases of training [8, 11].

Original: Understanding the role that data plays in these different phases can help us characterize what is being learned during them.
Revised: Understanding the role that data plays in these different phases can help us characterize what is being learned during them.

Original: Since data loading is often a bottleneck, this understanding also has the potential enable more efficient training schemes.
Revised: Since data loading is often a bottleneck, this understanding also has the potential enable more efﬁcient training schemes.

Original: In this work, we have considered two essential phases of early training: pre-training for IMP which enables sparse optimization and learning rate warmup which stabilizes network training.
Revised: In this work, we have considered two essential phases of early training: pre-training for IMP which enables sparse optimization and learning rate warmup which stabilizes network training.

Original: Our experiments identify what data is sufficient for each of these procedures, and as they both occur early in training, these finding can be used to design more efficient data loaders for streaming datasets.
Revised: Our experiments identify what data is sufﬁcient for each of these procedures, and as they both occur early in training, these ﬁnding can be used to design more efﬁcient data loaders for streaming datasets.

Original: Furthermore, as
Revised: Furthermore, as

Original: IMP is a computationally expensive procedure, understanding how it works is essential for designing better algorithms with the same performance.
Revised: IMP is a computationally expensive procedure, understanding how it works is essential for designing better algorithms with the same performance.

Original: To this end, we identify loss landscape properties of the dense network initialization for IMP that are predictive of successful spare training.
Revised: To this end, we identify loss landscape properties of the dense network initialization for IMP that are predictive of successful spare training.

Original: Though this work does not provide an improved algorithm for obtaining sparse networks, we believe our results provide essential guidance for researchers pursuing algorithms that perform pruning early in training (i.e. finding sparse masks without training to convergence).
Revised: Though this work does not provide an improved algorithm for obtaining sparse networks, we believe our results provide essential guidance for researchers pursuing algorithms that perform pruning early in training (i.e. ﬁnding sparse masks without training to convergence).

Original: Acknowledgements
Revised: Acknowledgements

Original: The experiments for this paper were funded by Google Cloud research credits.
Revised: The experiments for this paper were funded by Google Cloud research credits.

Original: S.G. thanks the James
Revised: S.G. thanks the James

Original: S. McDonnell and Simons Foundations, NTT Research, and an NSF CAREER Award for support while at Stanford.
Revised: S. McDonnell and Simons Foundations, NTT Research, and an NSF CAREER Award for support while at Stanford.

Original: This work was done in part while G.K.D. was visiting the Simons Institute for the Theory of Computing.
Revised: This work was done in part while G.K.D. was visiting the Simons Institute for the Theory of Computing.

Original: The authors would like to thank Daniel M. Roy for feedback on multiple drafts.A Experimental Details Code.
Revised: The authors would like to thank Daniel M. Roy for feedback on multiple drafts.

Original: The code used to run the experiments is available at: https://github.com/mansheej/ lth_diet

Original: Datasets.

Original: We used CIFAR-10, CIFAR-100 [18], and CINIC-10 [3] in our experiments.

Original: For CINIC10, we combine the training and validation sets into a single training set with 180,000 images.

Original: The standard test set of 90,000 images is used for testing.

Original: Each dataset is normalized by its per channel mean and standard deviation over the training set.

Original: All datasets get the same data augmentation: pad by 4 pixels on all sides, random crop to 32 × 32 pixels, and left-right flip image with probability half.

Original: Models.

Original: In these experiments we use ResNet-20, ResNet-32, ResNet-56 [15].

Original: These are the lowresolution CIFAR variants of ResNets from the original paper.

Original: The variants of the network used are specified in the figures.

Original: Randomized Labels.

Original: The labels were randomized during the pre-training phase only by first selecting 10%/50% of the training uniformly at random and then drawing a new label uniformly from the 10 classes of CIFAR-10.

Original: Note that because this procedure can result in an example being reassinged the correct label, on average only 9%/45% of the labels are corrupted by this procedure.

Original: The dataset is corrupted once and then resused across all subset sizes and replicates.

Original: The EL2N scores used to determine the subset of easiest data are computed by training on the corrupted dataset.

Original: As seen in Figure 5 of [1], the network typically learns the correct label early in training for corrupted data points.

Original: As a result, the corrupted examples will be ranked as difficult by the EL2N scores as verified in [23].

Original: Thus, the noisy labels are filtered out by pre-training on the examples with the lowest EL2N scores.

Original: Linear mode connectivity procedure.

Original: To determine if two children of a given network are linearly mode connected, we perform the following procedure:

Original: Spawn two children initialized with the current weights of the given network.

Original: Independently train the two networks with different SGD noise realized through different data orders.

Original: Calculate the mid-point in weight space between the two networks.

Original: Calculate the training loss for the resulting network at the mid-point.

Original: Calculate the training loss barrier—the loss calculated at the mid-point minus the mean of the losses calculated at the endpoints.

Original: Train loss barriers are calculated both on a per-example basis and as averages over the whole dataset.

Original: The per-example training loss barrier is just the training loss barrier evaluated on that example.

Original: In practice, we train three children and calculate the train loss barrier between each pair and average.

Original: Following [10], we say that two networks are linearly mode connected if the train loss barrier between them is less than 2%.

Original: EL2N score computation To calculate EL2N scores for a dataset, we follow the process outlined in [23].

Original: In particular, we do the following:

Original: Independently train K = 10 networks from different random initializations for

Original: t

Original: iterations.

Original: For each example and each network, we calculate the L2 norm of the error vector defined as ∥ p ( x ) − y ∥ 2 where y is the one-hot encoding of the label, and p ( x ) are the softmax outputs of the network evaluated on example x . 3.

Original: For each example, the EL2N score is the average of the error vector L2 norm across the K networks.

Original: To calculate these scores, we use ResNet-20 and t = 7800 iterations for CIFAR-10, ResNet-32 and t = 7800 iterations for CIFAR-100, and ResNet-56 and t = 8000 iterations for CINIC-10.

Original: Hyperparameters.

Original: Networks were trained with stochastic gradient descent (SGD).

Original: t ∗ was chosen for each dataset such that t r = t ∗ produces sparse networks that can be trained to the same accuracy as the dense network to at least 16.8% sparsity (8 pruning levels).

Original: The pre-training learning rate was chosen based on which of the set { 0 . 1 , 0 . 2 , 0 . 4 } produced the best performance at t ∗ .

Original: The full hyperparameters are provided in Table 1.

Original: [Table]

Original: All results are reported as the mean and standard deviation of 4 replicates.

Original: The only exception is

Original: Figure 5 scatter plots where each point corresponds to 1 replicate.

Original: Learning rate warmup experimental design.

Original: For the learning rate warm-up experiments, we train

Original: ResNet-20 on CIFAR-10 with batch size = 1024.

Original: For the optimizer, we use SGD with learning rate = 3.2, momentum = 0.9 and weight decay = 0.0001.

Original: There is an initial linear learning rate warm-up for t batches.

Original: The network is then trained for a total of 160 epochs with a LR drop by a factor of 10 atand 120 epochs.

Original: In Figure 6, we sweep the learning rate warmup period t .

Original: For each hyperparameter configuration, we report the mean and standard deviation of 8 replicates.

Original: In addition to all the training data, we perform experiments with different subsets of the training data with sizes 5120, 10240 and 25600.

Original: In these cases, the network was trained on the data subset during the warm-up period only and on the full dataset afterwards.

Original: For the easiest (hardest) examples, we choose examples with the lowest (highest) EL2N scores which were calculated as described above.

Original: Compute Resources.

Original: The experiments were performed on virtual Google Cloud instances configured with 4 NVIDIA Tesla A100 GPUs.

Original: Each experiment replicate was run on a single A

Original: GPU.

Original: The approximate compute time for a full run of IMP was 8 hours for CIFAR-10, 12 hours for

Original: CIFAR-100, and 10 hours for CINIC-10.

Original: Ethical and societal consequences.

Original: Being empirically-driven work, the experiments performed consumed considerable energy.

Original: However, we view our work as a step towards the long-term goal of improving the efficiency of training neural network which will ideally lead to an increase in the democratization and sustainability of AI.

Original: Towards this aim, our paper provides insight into the role of data in early training and a better understanding of which properties of pre-trained initializations enable sparse optimization.

Original: We hope this will inspire better algorithms for training sparse networks.

Original: B Explanation of the Role of Easy Data

Original: Our primary observation in Section 3 is that training on a small fraction of easy data can reduce the number of steps needed to find an initialization which contains a matching sparse network, i.e. one that can be trained to the same accuracy as the dense network.

Original: An explanation for this phenomenon can be divided into two questions.

Original: First, what characteristic of a pre-trained dense network can predict if it contains a matching sparse network?

Original: Second, does training only on easy examples allow us to find networks with this characteristic faster and why?

Original: In Section 4, we try to provide a linear mode connectivity (LMC)

Original: based explanation for the first question.

Original: Previous work [10] has shown that the rewind step at which sparse networks become matching correlates with the onset of LMC for the sparse network.

Original: Intuitively, this suggests that a notion of stability is important for finding matching sparse networks—such networks exist at the

Original: [Figure]

Original: rewind step at which the sparse network optimization becomes stable to SGD noise.

Original: However, this occurs before the onset of LMC in the dense network.

Original: Thus, we generalized the notion of stability to

Original: SGD noise from a binary metric—is the train loss barrier below some threshold—to a continuous one—the average train loss barrier—and found that this characteristic of the dense network is indeed correlated with our ability to find matching initializations.

Original: To answer the second question, we show that training only on easy examples finds networks with lower average train loss barriers faster and the resulting distribution of per-example train loss barriers is very similar to those obtained from training on all data for twice as many steps (Section 4, Figure 5).

Original: Here we further investigate why this might be the case.

Original: First, we compare how fast networks learn when trained on easy vs. all data.

Original: For CIFAR-100 + ResNet-32, we train for 1600 steps on both a subset of 2048 easy examples and on all examples.

Original: We measure the test accuracy as well as the train accuracy on both the full dataset and the easy subset.

Original: The results in Figure 7a show that training on easy examples performs better on all 3 metrics, leading to faster learning.

Original: Next we compare the coherence of gradients along these training trajectories.

Original: Every 10 steps, we sample 16 batches from the dataset used to train the network and calculate the mini-batch gradients.

Original: For each step, we then measure the cosine similarity between each sampled gradient and the mean gradient across batches.

Original: The results in Figure 7b show that this similarity is consistently higher when training on easy data.

Original: Previous work has shown that networks learn easy examples before learning hard examples [21].

Original: Our results further show that, very early in training, making repeated passes through the easy data creates a stronger and more coherent training signal which leads to faster learning.

Original: An intuitive summary of these results is, given that the information necessary for finding matching subnetworks is learned in the first few hundred steps, it is more efficient to focus training resources on examples that can be learned well early in training.

Original: C Other Hypotheses for the Role of Easy Data: Negative Results

Original: Pre-training on easy data (lowest EL2N scores) allows us to reduce t r without hurting test accuracy of the sparse networks found by IMP.

Original: One possible reason for this is that networks pre-trained on just the easy data achieve better test accuracies at step t r .

Original: But in Figure 5, we find that the accuracy of the dense network at the pre-trained initialization does not completely explain the improved performance.

Original: Here we explore two additional hypotheses we considered for what role the easy data plays during the pre-training phase that we found to be not explanatory.

Original: C.1 Changes in the gradient size.

Original: Train a ResNet-20 on CIFAR-10 for 200 iterations on a random subset of 25600 examples.

Original: At every iteration record the gradient norm of the batch.

Original: Repeat the above process but training on a subset of 25600 examples with the smallest EL2N scores.

Original: Compare the distribution of gradient norms

Original: Examples with lower EL2N scores have smaller gradient norms at the point at which they are computed as observed by [23].

Original: We investigate whether the gradient norm is also smaller when training on the easiest examples during the pre-training phase of IMP, which would reproduce the effects of gradient clipping which is often beneficial early in training.

Original: We compare the distribution of batch gradient norms seen in the first 200 steps while training on easy data versus a random subset of data.

Original: Note that since we are only training for 200 steps with a batch size of 128, training on all the data is equivalent to training on random data.

Original: We run the following experiment:

Original: Figure 8 shows the results.

Original: We find that, when training on easy examples, the batch gradient norms are typically larger.

Original: Our hypothesis is that on easy examples, the gradients are more aligned, leading to larger average gradients.

Original: We thus reject the hypothesis that pre-training on the easy subset of the data effectively performs gradient clipping.

Original: C.2 Change in distribution of per-example error

Original: [Figure]

Original: For a ResNet-56 trained on CINIC-10, we considered whether different pre-training procedures resulted in diffferent distributions of the per-example error early in training.

Original: We made the following observations in the results shown in Figure

Original: Prior to the sparse projection, there is a distinctive distribution of error after training on easy data.

Original: A split occurs in the data where a portion of the examples shifts towards zero error while the remainder increases in error as compared to training on all data (level 0 green curve vs. level 0 blue curve in Figure 9).

Original: When comparing the dense vs. sparse networks pre-trained on all data, the distribution looks the same before and after projection (level 0 vs. level 6 blue and orange curves).

Original: However, for pre-training on the easy data a significant change happens after the projection (level 0 vs. levelgreen curves).

Original: After the projection, the error distributions from training on all data for 800 iterations and easy data for 400 iterations are similar (level 6 orange and green curve) while the training on all data for 400 iterations is different (level 6 blue curve).

Original: These observations bear similarity to the per-example train loss barrier results considered in the main text (Figure 5).

Original: Pre-training on easy data for 400 iterations produces results that have the same signatures as pre-training on all data for 800 iterations (and differ from training on all data foriterations).

Original: However, the per-example train loss barrier provides a clearer explanation and thus we think is the more fundamental quantity.

Original: D Linear Mode Connetivity: Additional Experiments

Original: Here we present additional work on understanding the connection between linear mode connectivity of a dense initialization and its performance as a pre-trained initialization for IMP.

Original: We introduce a new score for ranking examples in terms of the per-example train loss barrier early in training which we call the LMC score.

Original: We then compare this score to EL2N and test the IMP performance of pre-training on the subset of examples with low LMC scores.

Original: D.1 Background on (linear) mode connectivity.

Original: The deep learning phenomena of mode connectivity via simple paths has been uncovered by Draxler et al.

Original: [5], Garipov et al.

Original: In particular, it was shown that SGD-trained deep neural network modes are connected via simple nonlinear paths, along which the training and test loss is approximately constant.

Original: Since then this phenomena has also been studies theoretically [19].

Original: A related mode property has been described in the literature on generalization in deep learning: Nagarajan and Kolter [22] show that for small networks trained on MNIST there exist linearly connected modes.

Original: Surprisingly, a slightly modified version of this phenomena is also true for large scale vision networks: Frankle et al.

Original: [10] show that coupling the first training epochs restricts SGD to converge to the same linearly connected mode.

Original: More recently, Entezari et al.

Original: [6] hypothesize that linearly connected modes are equivalent up to symmetries.

Original: Their empirical observations suggest that there is overlap between the functions parameterized by different modes.

Original: D.2 LMC Scores

Original: Our observation that successful pre-train initializations are correlated with a shift to smaller perexample train loss barriers (left and center columns of Figure 5) naturally raises the question: are examples that become linearly mode connected first also the ones which are important for this phase of training?

Original: To this end, we introduced the notion of a per-example LMC score which allows us to rank examples in this manner:

Original: Definition D.1 The LMC score of an example is the loss barrier computed on a dense network at rewinding iteration t averaged over K instances.

Original: Under this definition, the train error barrier is the LMC scores averaged over the training data.

Original: If we compute the score after the onset of linear mode connectivity ¯ t it will not provide an informative ranking of the data as the score will be close to 0 for all examples; we compute the scores before ¯ t .

Original: [Figure]

Original: [Figure]

Original: Several of runs are included in the labelled

Original: [Figure]

Original: In our experiments

Original: [Figure]

Original: we computed the LMC score by averaging over three children runs from two replicates, so that each replicate provided three train loss barriers (one for each pair of children) and the total average was taken over six instances.

Original: We visualize the correlation between EL2N scores and LMC scores.

Original: The scatter plots in Figure

Original: Figure 10 show that low EL2N score examples tend to be the ones for which the LMC score goes down rapidly with spawning time, and is nearly-zero early in training.

Original: However, the opposite is not true, and high EL2N scores have a wide range of per-example loss barriers.

Original: D.3 Pre-training on Low LMC Scores

Original: Figure 11 shows that rewinding points obtained by training on low LMC score examples dominate random example pre-training, but are (weakly) dominated by training on low EL2N score examples.

Original: We thus conclude that low EL2N examples are more representative of the necessary and sufficient subset of examples needed to find a good rewinding point.

Original: E Full Results E.1 Comparison to Forgetting Scores

Original: [Figure]

Original: In Figure 12, we compare pre-training on the smallest EL2N scores vs. smallest forgetting scores.

Original: The forgetting score of an example is defined by Toneva et al.

Original: [24] as the number of times the example is included in a mini-batch and the network switches from a correct to incorrect classification of that example—a “forgetting event”.

Original: As seen in the results, pre-training on the subset of smallest forgetting scores underperforms using the subset of smallest EL2N scores computed early in training for subset sizes 1024, 2048, and 3200.

Original: There are two reasons why we might expect forgetting scores to underperform EL2N scores.

Original: First, forgetting scores as defined here look at example difficulty over the whole course of training whereas we computed the EL2N scores early in training at 16 epochs.

Original: For IMP pre-training, we are most interested in finding examples which the network finds easy early in training.

Original: [-Second, for many of the easy examples the forgetting score is 
-]
Revised: [-Second, for many of the easy examples the forgetting score is 
-]

Original: meaning the example is learned once and never forgotten.

Original: If this is true for a large number of examples, this ranking is less useful when trying to choose a small subset size (for both scores, the subset was class-balanced).

Original: Our goal was to identify training data subsets that suffice for pre-training.

Original: Our choice of metric was thus intentional: EL2N scores capture how difficult it is to learn an example early in training, which intuitively correlates to stability of training and error barriers.

Original: E.2 Results Across Subset Size

Original: Here we present the full set of experiments performed for the results in the main text, sweeping across subset size.

Original: Figure 13 and Figure 14 show the results for performing IMP pre-training with different data subsets across a range of different subset sizes.

Original: Figure 15 shows the result for performing IMP pre-training with the dataset corrupted by randomized label noise (the original dataset is then used for the mask search and sparse training phase).

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: l s 10 % R a n d o m i

Original: z e d L a b e l s