Original: D RAWING EARLY - BIRD TICKETS : T OWARDS MORE EFFICIENT TRAINING OF DEEP NETWORKS
Revised: D RAWING EARLY - BIRD TICKETS : T OWARDS MORE EFFICIENT TRAINING OF DEEP NETWORKS

Original: Anonymous authors Paper under double-blind review
Revised: A BSTRACT

Original: A BSTRACT (Frankle & Carbin, 2019) shows that there exist winning tickets (small but critical subnetworks) for dense, randomly initialized networks, that can be trained alone to achieve comparable accuracies to the latter in a similar number of iterations.
Revised: (Frankle & Carbin, 2019) shows that there exist winning tickets (small but critical subnetworks) for dense, randomly initialized networks, that can be trained alone to achieve comparable accuracies to the latter in a similar number of iterations.

Original: However, the identiﬁcation of these winning tickets still requires the costly trainprune-retrain process, limiting their practical beneﬁts.
Revised: However, the identiﬁcation of these winning tickets still requires the costly trainprune-retrain process, limiting their practical beneﬁts.

Original: In this paper, we discover for the ﬁrst time that the winning tickets can be identiﬁed at the very early training stage , which we term as early-bird (EB) tickets, via low-cost training schemes (e.g., early stopping, low-precision training) at large learning rates.
Revised: In this paper, we discover for the ﬁrst time that the winning tickets can be identiﬁed at the very early training stage , which we term as Early-Bird (EB) tickets, via low-cost train- ing schemes (e.g., early stopping, low-precision training) at large learning rates.

Original: Our ﬁnding of EB tickets is consistent with recently reported observations that the key connectivity patterns of neural networks emerge early.
Revised: Our ﬁnding of EB tickets is consistent with recently reported observations that the key connectivity patterns of neural networks emerge early.

Original: Furthermore, we propose a mask distance metric that can be used to identify EB tickets with low computational overhead, without needing to know the true winning tickets that emerge after the full training.
Revised: Furthermore, we propose a mask distance metric that can be used to identify EB tickets with low computational overhead, without needing to know the true winning tickets that emerge after the full training.

Original: Finally, we leverage the existence of EB tickets and the proposed mask distance to develop efﬁcient training methods, which are achieved by ﬁrst identifying EB tickets via low-cost schemes, and then continuing to train merely the EB tickets towards the target accuracy.
Revised: Finally, we leverage the existence of EB tickets and the proposed mask distance to develop efﬁcient training methods, which are achieved by ﬁrst identifying EB tickets via low-cost schemes, and then continuing to train merely the EB tickets towards the target accuracy.

Original: Experiments based on various deep networks and datasets validate: 1) the existence of EB tickets, and the effectiveness of mask distance in efﬁciently identifying them; and 2) that the proposed efﬁcient training via EB tickets can achieve up to 4.7 × energy savings while maintaining comparable or even better accuracy, demonstrating a promising and easily adopted method for tackling cost-prohibitive deep network training. 
Revised: Experiments based on various deep networks and datasets validate: 1) the existence of EB tickets, and the effectiveness of mask distance in efﬁciently identifying them; and 2) that the proposed efﬁcient training via EB tickets can achieve up to 4.7 × energy savings while maintaining comparable or even better accuracy, demonstrating a promising and easily adopted method for tackling cost-prohibitive deep network training. Code available at https://github.com/RICE-EIC/Early-Bird-Tickets

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The recent record-breaking predictive performance achieved by deep neural networks (DNNs) motivates a tremendously growing demand to bring DNN-powered intelligence into numerous applications.
Revised: The recent record-breaking predictive performance achieved by deep neural networks (DNNs) motivates a tremendously growing demand to bring DNN-powered intelligence into numerous applications.

Original: However, the excellent performance of modern DNNs comes at an often prohibitive training cost due to the required vast volume of training data and model parameters.
Revised: However, the excellent performance of modern DNNs comes at an often prohibitive training cost due to the required vast volume of training data and model parameters.

Original: As an illustrative example of the computational complexity of DNN training, one forward pass of the ResNet50 (He et al.
Revised: As an illustrative example of the computational complexity of DNN training, one forward pass of the ResNet50 (He et al.

Original: (2016a)) model requires 4 GFLOPs (FLOPs: ﬂoating point operations per second) of computations and training requires 10 18 FLOPs, which takes 14 days on one state-of-the-art
Revised: (2016a)) model requires 4 GFLOPs (FLOPs: ﬂoating-point operations) of computations and training requires 10 18 FLOPs, which takes 14 days on one state-of-the-art

Original: NVIDIA M40 GPU (You et al.
Revised: NVIDIA M40 GPU (You et al.

Original: As a result, training a state-of-the-art DNN model often demands considerable energy, along with the associated ﬁnancial and environmental costs.
Revised: As a result, training a state-of-the-art DNN model often demands considerable energy, along with the associated ﬁnancial and environmental costs.

Original: For example, a recent report shows that training a single DNN can cost over $10K US dollars and emit as much carbon as ﬁve cars in their lifetimes (Strubell et al.
Revised: For example, a recent report shows that training a single DNN can cost over $10K US dollars and emit as much carbon as ﬁve cars in their lifetimes (Strubell et al.

Original: , limiting the rapid development of DNN innovations and raising various environmental concerns.
Revised: , limiting the rapid development of DNN innovations and raising various environmental concerns.

Original: The recent trends of improving DNN efﬁciencyare mostly focused on compressing models and accelerating inference.
Revised: The recent trends of improving DNN efﬁciency mostly focus on compressing models and accelerating inference.

Original: An empirically adopted practice is the so-called progressive pruning and training routine, i.e., training a large model fully, pruning it and then retraining the pruned model to restore the performance (the process can be iterated several rounds).
Revised: An empirically adopted practice is the so-called progressive pruning and training routine, i.e., training a large model fully, pruning it and then retraining the pruned model to restore the performance (the process can be iterated several rounds).

Original: While this has been a standard practice for model compression (Han et al., 2015), some recent efforts start empirically linking it to the potential of more efﬁcient training.
Revised: While this has been a standard practice for model compression (Han et al., 2015), some recent efforts start empirically linking it to the potential of more efﬁcient training.

Original: Notably, a latest series of works (Frankle & Carbin, 2019; Liu et al., 2018) reveals that dense, randomly-initialized networks contain small subnetworks which can match the test accuracy of original networks when trained alone themselves.
Revised: Notably, the latest series of works (Frankle & Carbin, 2019; Liu et al., 2018) reveals that dense, randomly-initialized networks contain small subnetworks which can match the test accuracy of original networks when trained alone themselves.

Original: These subnetworks are called winning tickets .
Revised: These subnetworks are called winning tickets .

Original: Despite their insightful ﬁndings, there remains to be a major gap between the winning ticket observation and the goal of more efﬁcient training, since winning tickets were only identiﬁed by pruning unimportant connections in a fully trained dense network.
Revised: Despite their insightful ﬁndings, there remains to be a major gap between the winning ticket observation and the goal of more efﬁcient training, since winning tickets were only identiﬁed by pruning unimportant connections in a fully trained dense network.

Original: This paper closes this gap by demonstrating the Early-Bird (EB) tickets phenomenon: the winning tickets can be drawn very early in training, and with aggressively low-cost training algorithms .
Revised: This paper closes this gap by demonstrating the Early-Bird (EB) tickets phenomenon: the winning tickets can be drawn very early in training, and with aggressively low-cost training algorithms .

Original: Through a range of experiments on different DNNs and datasets, we observe the consistent existence of EB tickets, the cheap costs needed to reliably draw them, and develop a novel mask distance metric to detect their emergence.
Revised: Through a range of experiments on different DNNs and datasets, we observe the consistent existence of EB tickets, the cheap costs needed to reliably draw them, and develop a novel mask distance metric to detect their emergence.

Original: After being identiﬁed, re-training those EB-tickets (using standard training) leads to comparable or even superior ﬁnal accuracies, compared to either standard training, or re-training the “ground-truth” winning tickets drawn after full training as in (Frankle & Carbin, 2019).
Revised: After being identiﬁed, re-training those EB tickets (using standard training) leads to comparable or even superior ﬁnal accuracies, compared to either standard training, or re-training the “ground-truth” winning tickets drawn after full training as in (Frankle & Carbin, 2019).

Original: Our observations seem to coincide with the recent ﬁndings by (Achille et al., 2019; Li et al., 2019) about the two-stage optimization trajectory in training.
Revised: Our observations seem to coincide with the recent ﬁndings by (Achille et al., 2019; Li et al., 2019) about the two-stage optimization trajectory in training.

Original: Taking advantage of EB tickets, we propose an efﬁcient DNN training scheme termed EB Train .
Revised: Taking advantage of EB tickets, we propose an efﬁcient DNN training scheme termed EB Train .

Original: To our best knowledge, this is the ﬁrst step taken towards exploiting winning tickets for a realistic efﬁcient training goal.
Revised: To our best knowledge, this is the ﬁrst step taken towards exploiting winning tickets for a realistic efﬁcient training goal.

Original: Our contribution can be summarized as follow:
Revised: Our contribution can be summarized as follow:

Original: We discover the Early-Bird (EB) tickets, and show they 1) consistently exist across models and datasets; 2) can emerge very early in training; and 3) stay robust under (and sometimes even favor) various aggressive and low-cost training schemes (in addition to early stopping), including large learning rates and low-precision training.
Revised: We discover the Early-Bird (EB) tickets, and show they 1) consistently exist across models and datasets; 2) can emerge very early in training; and 3) stay robust under (and sometimes even favor) various aggressive and low-cost training schemes (in addition to early stopping), including large learning rates and low-precision training.

Original: We propose a practical, easy-to-compute mask distance, as an indicator informing to draw EB tickets without accessing the “ground-truth” winning tickets (drawn after full training), ﬁxing a major paradox for connecting winning tickets with the efﬁcient training goal.
Revised: We propose a practical, easy-to-compute mask distance, as an indicator informing to draw EB tickets without accessing the “ground-truth” winning tickets (drawn after full training), ﬁxing a major paradox for connecting winning tickets with the efﬁcient training goal.

Original: We design a novel efﬁcient training framework based on EB tickets (EB Train).
Revised: We design a novel efﬁcient training framework based on EB tickets (EB Train).

Original: Experiments in state-of-the-art benchmarks and models show that EB Train can achieve up to 4.7 × energy savings , while maintaining the same or even better accuracy, compared to training with the original winning tickets.
Revised: Experiments in state-of-the-art benchmarks and models show that EB Train can achieve up to 4.7 × energy savings , while maintaining the same or even better accuracy, compared to training with the original winning tickets.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Winning Ticket Hypothesis.
Revised: Winning Ticket Hypothesis.

Original: The lottery ticket hypothesis (Frankle & Carbin, 2019)
Revised: The lottery ticket hypothesis (Frankle & Carbin, 2019)

Original: ﬁrst points out that a small subnetwork, called the winning ticket, can be identiﬁed by pruning a full trained dense network; when training it in isolation with the same weight initialization once assigned to the corresponding weights in the dense network, one can restore the comparable test accuracy to the dense network.
Revised: ﬁrst points out that a small subnetwork, called the winning ticket, can be identiﬁed by pruning a full trained dense network; when training it in isolation with the same weight initialization once assigned to the corresponding weights in the dense network, one can restore the comparable test accuracy to the dense network.

Original: However, ﬁnding winning tickets hinged on costly (iterative) pruning and retraining (Morcos et al., 2019) studies the reuse of winning tickets, transferable across different datasets.
Revised: However, ﬁnding winning tickets hinged on costly (iterative) pruning and retraining. (Morcos et al., 2019) studies the reuse of winning tickets, transferable across different datasets.

Original: (Zhou et al., 2019) discovers the existence of supermasks that can be applied to an untrained, randomly-initialized network.
Revised: (Zhou et al., 2019) discovers the existence of supermasks that can be applied to an untrained, randomly-initialized network.

Original: (Liu et al., 2018) argues that the weight initialization might make less difference when trained with a large learning rate, while the searched connectivity is more of the winning ticket’s core value.
Revised: (Liu et al., 2018) argues that the weight initialization might make less difference when trained with a large learning rate, while the searched connectivity is more of the winning ticket’s core value.

Original: It also explores the usage of both unstructured and (more hardware-friendly) structured pruning and shows that both lead to the emergence of winning tickets.
Revised: It also explores the usage of both unstructured and (more hardware-friendly) structured pruning and shows that both lead to the emergence of winning tickets.

Original: Another related work (Lee et al., 2019) prunes a network at single-shot with one mini-batch, in which the irrelevant connections are identiﬁed by a connection sensitivity criterion.
Revised: Another related work (Lee et al., 2019) prunes a network at single-shot with one mini-batch, in which the irrelevant connections are identiﬁed by a connection sensitivity criterion.

Original: Comparing to (Frankle & Carbin, 2019), the authors showed their method to be more efﬁcient in ﬁnding the good subnetwork (not the winning ticket), although its re-training accuracy/efﬁciency was found to be inferior, compared to training the “ground truth” winning ticket.
Revised: Comparing to (Frankle & Carbin, 2019), the authors show their method to be more efﬁcient in ﬁnding the good subnetwork (not the winning ticket), although its re-training accuracy/efﬁciency is found to be inferior, compared to training the “ground truth” winning ticket.

Original: Other Relevant Observations in Training.
Revised: Other Relevant Observations in Training.

Original: (Rahaman et al., 2019; Xu et al., 2019) argue that deep networks will ﬁrst learn low-complexity (lower-frequency) functional components, before absorbing high-frequency features: the former being more robust to perturbations.
Revised: (Rahaman et al., 2019; Xu et al., 2019) argue that deep networks will ﬁrst learn low-complexity (lower-frequency) functional components, before absorbing high-frequency features: the former being more robust to perturbations.

Original: An important hint can be found in (Achille et al., 2019): the early stage of training seems to ﬁrst discover the important connections and the connectivity patterns between layers, which becomes relatively ﬁxed in the latter training stage.
Revised: An important hint can be found in (Achille et al., 2019): the early stage of training seems to ﬁrst discover the important connections and the connectivity patterns between layers, which becomes relatively ﬁxed in the latter training stage.

Original: That seems to imply that the critical sub-network (connectivity) can be identiﬁed independent of, and seemingly also ahead of, the (ﬁnal best) weights.
Revised: That seems to imply that the critical sub-network (connectivity) can be identiﬁed independent of, and seemingly also ahead of, the (ﬁnal best) weights.

Original: Finally, Li et al.
Revised: Finally, Li et al.

Original: (2019) demonstrates that training a deep network with a large initial learning rate helps the model focus on memorizing easier-to-ﬁt, more generalizable pattern faster and better – a direct inspiration for us to try drawing EB tickets using large learning rates.
Revised: (2019) demonstrates that training a deep network with a large initial learning rate helps the model focus on memorizing easier-to-ﬁt, more generalizable pattern faster and better – a direct inspiration for us to try drawing EB tickets using large learning rates.

Original: Efﬁcient Inference and Training.
Revised: Efﬁcient Inference and Training.

Original: Model compression has been extensively studied for lighterweight inference.
Revised: Model compression has been extensively studied for lighterweight inference.

Original: Popular means include pruning (Li et al., 2017;
Revised: Popular means include pruning (Li et al., 2017;

Original: Liu et al., 2017; He et al., 2018;
Revised: Liu et al., 2017; He et al., 2018;

Original: Wen et al., 2016; Luo et al., 2017), weight factorization (Denton et al., 2014), weight sharing (Wu et al., 2018a), quantization (Hubara et al., 2017), and lately network architecture search (Zoph & Le, 2017).
Revised: Wen et al., 2016; Luo et al., 2017), weight factorization (Denton et al., 2014), weight sharing (Wu et al., 2018a), quantization (Hubara et al., 2017), dynamic inference (Wang et al., 2018b; 2019b), network architecture search (Zoph & Le, 2017), among many others (Wang et al., 2018c;d).

Original: On the other hand, the literature on efﬁcient training appears to be much sparser.
Revised: On the other hand, the literature on efﬁcient training appears to be much sparser.

Original: A handful of works (Goyal et al., 2017; Cho et al., 2017; You et al., 2018; Akiba et al., 2017; Jia et al., 2018; Gupta et al., 2015) focus on reducing the total training time in paralleled, communication-efﬁcient distributed settings.
Revised: A handful of works (Goyal et al., 2017; Cho et al., 2017; You et al., 2018; Akiba et al., 2017; Jia et al., 2018; Gupta et al., 2015) focus on reducing the total training time in paralleled, communication-efﬁcient distributed settings.

Original: In contrast, our goal is to shrink the total resource cost for in-situ, resourceconstrained training.
Revised: In contrast, our goal is to shrink the total resource cost for in-situ, resource-constrained training, as (Wang et al., 2019a) advocated.

Original: (Banner et al., 2018; Wang et al., 2018) presented low-precision training, which is aligned with our goal and can be incorporated into EB Train (see later).
Revised: (Banner et al., 2018; Wang et al., 2018a) presented low-precision training, which is aligned with our goal and can be incorporated into EB Train (see later).

Original:  D RAWING E ARLY - BIRD T ICKETS : H YPOTHESIS AND E XPERIMENTS
Revised:  D RAWING E ARLY - BIRD T ICKETS : H YPOTHESIS AND E XPERIMENTS

Original: We hypothesize that the winning tickets can emerge at a very early training stage , which we term as an early-bird (EB) ticket.
Revised: We hypothesize that the winning tickets can emerge at a very early training stage , which we term as an Early-Bird (EB) ticket.

Original: Consider a dense, randomly-initialized network f ( x ; θ ) , f reaches a minimum validation loss f loss at the i -th iteration with a test accuracy f acc , when optimized with stochastic gradient descent (SGD) on a training set.
Revised: Consider a dense, randomly-initialized network f ( x ; θ ) , f reaches a minimum validation loss f loss at the i -th iteration with a test accuracy f acc , when optimized with stochastic gradient descent (SGD) on a training set.

Original: In addition, consider subnetworks f ( x ; m (cid:12) θ ) with a mask m ∈ { 0 , 1 } indicates the pruned and unpruned connections in f ( x ; θ ) .
Revised: In addition, consider subnetworks f ( x ; m (cid:12) θ ) with a mask m ∈ { 0 , 1 } indicates the pruned and unpruned connections in f ( x ; θ ) .

Original: When being optimized with SGD on the same training set, f ( x ; m (cid:12) θ ) reach a minimum validation loss f (cid:48) loss at the i (cid:48)
Revised: When being optimized with SGD on the same training set, f ( x ; m (cid:12) θ ) reach a minimum validation loss f (cid:48) loss at the i (cid:48)

Original: -th iteration with a test accuracy f (cid:48) acc .
Revised: -th iteration with a test accuracy f (cid:48) acc .

Original: The EB tickets hypothesis articulates that there exists m such
Revised: The EB tickets hypothesis articulates that there exists m such

Original: that f (cid:48) acc ≈ f acc (even ≥ ), i.e., same or better generalization, with i (cid:48)
Revised: that f (cid:48) acc ≈ f acc (even ≥ ), i.e., same or better generalization, with i (cid:48)

Original: (cid:28) i
Revised: (cid:28) i

Original: (e.g., early stopping) and sparse m (i.e., much reduced parameters).
Revised: (e.g., early stopping) and sparse m (i.e., much reduced parameters).

Original: Section 3 addresses three key questions pertaining to the EB ticket hypothesis.
Revised: Section 3 addresses three key questions pertaining to the EB ticket hypothesis.

Original: We ﬁrst show via an extensive set of experiments, that EB tickets can be observed across popular models and datasets (Section 3.1).
Revised: We ﬁrst show via an extensive set of experiments, that EB tickets can be observed across popular models and datasets (Section 3.1).

Original: We then try to be more aggressive to see if high-quality EB tickets still emerge under “cheaper” training (Section 3.2).
Revised: We then try to be more aggressive to see if high-quality EB tickets still emerge under “cheaper” training (Section 3.2).

Original: We ﬁnally reveal that EB tickets can be identiﬁed with comparing a novel mask distance between consecutive epochs, thus no full training needed (Section 3.3).
Revised: We ﬁnally reveal that EB tickets can be identiﬁed with comparing a novel mask distance between consecutive epochs, thus no full training needed (Section 3.3).

Original: 3. D O E ARLY - BIRD T ICKETS A LWAYS E XIST ?
Revised: D O E ARLY - BIRD T ICKETS A LWAYS E XIST ?

Original: We perform ablation simulations using two representative deep models: VGG16 (Simonyan & Zisserman, 2014) and pre-activation residual networks-101 (PreResNet101) (He et al., 2016b), on two popular datasets: CIFAR-10 and CIFAR-100.
Revised: We perform ablation simulations using two representative deep models: VGG16 (Simonyan & Zisserman, 2014) and pre-activation residual networks-101 (PreResNet101) (He et al., 2016b), on two popular datasets: CIFAR-10 and CIFAR-100.

Original: For drawing the tickets (submodels), we adopt a standard training protocol  for both CIFAR-10 and CIFAR-100: the training takes 160 epochs in total and the batch size to 256; the initial learning rate is set to 0.1, and is divided by 10 at the 80th and 120th epochs respectively; the SGD solver is adopted with a momentum of 0.9 and a weight decay of 10 − 4 .
Revised: For drawing the tickets, we adopt a standard training protocol (Liu et al., 2018) for both CIFAR-10 and CIFAR-100: the training takes 160 epochs in total and the batch size to 256; the initial learning rate is set to 0.1, and is divided by 10 at the 80th and 120th epochs respectively; the SGD solver is adopted with a momentum of 0.9 and a weight decay of 10 − 4 .

Original: For retraining the tickets, we keep the same setting by default.
Revised: For retraining the tickets, we keep the same setting by default.

Original: We follow the main idea of (Frankle & Carbin, 2019), but instead prune networks trained at earlier points (before the accuracies reach their ﬁnal top values), to see if reliable tickets can still be drawn.
Revised: We follow the main idea of (Frankle & Carbin, 2019), but instead prune networks trained at earlier points (before the accuracies reach their ﬁnal top values), to see if reliable tickets can still be drawn.

Original: We adopt the same channel pruning in (Liu et al., 2017) for all experiments since it is hardware friendly and aligns with our end goal of efﬁcient training (Wen et al., 2016).
Revised: We adopt the same channel pruning in (Liu et al., 2017) for all experiments since it is hardware friendly and aligns with our end goal of efﬁcient training (Wen et al., 2016).

Original: Fig.
Revised: Fig.

Original: 1 reports the accuracies achieved by re-training the tickets drawn from different early epochs.
Revised: 1 reports the accuracies achieved by re-training the tickets drawn from different early epochs.

Original: All results consistently endorse that there exist high-quality tickets, at as early as epoch 20 (w.r.t.
Revised: All results consistently endorse that there exist high-quality tickets, at as early as epoch 20 (w.r.t.

Original: totally 160 epochs), that can achieve strong re-training accuracies.
Revised: totally 160 epochs), that can achieve strong re-training accuracies.

Original: Comparing among different pruning ratios p , it is not too surprising to see over-pruning (e.g., p = 70%) makes drawing good tickets harder, indicating a balance that we need to calibrate between accuracy and training efﬁciency.
Revised: Comparing among different pruning ratios p , it is not too surprising to see over-pruning (e.g., p = 70%) makes drawing good tickets harder, indicating a balance that we need to calibrate between accuracy and training efﬁciency.

Original: Two more striking observations from Fig.
Revised: Two more striking observations from Fig.

Original: 1: 1) there consistently exist EB tickets drawn at certain early epoch ranges, that outperform those drawn in later stages, including the “ground-truth” winning tickets drawn at epoch 160.
Revised: 1: 1) there consistently exist EB tickets drawn at certain early epoch ranges, that outperform those drawn in later stages, including the “ground-truth” winning tickets drawn at epoch 160.

Original: That intriguing phenomenon implies the possible “over-cooking” when networks try to identify connectivity patterns ﬁrst (Achille et al., 2019), that might hampergeneration; 2) some EB tickets are able to outperform even their unpruned, fully-trained models (e.g., the dashlines), potentially thanks to the sparse regularization learned by EB tickets. 3. D O E ARLY - BIRD T ICKETS S TILL E MERGE UNDER L OWER -C OST T RAINING ? For EB tickets, only the important connections found by them (connectivity) matters, while the weights are to be re-trained anyway. One might hence come up with an idea, that more aggressive and “cheaper” training methods might be applicable to further shrink the cost of ﬁnding EB tickets (which already implies early stopping of training), as long as the same signiﬁcant connections still emerge.
Revised: That intriguing phenomenon implies the possible “over-cooking” when networks try to identify connectivity patterns ﬁrst (Achille et al., 2019), that might hamper

Original: We experimentally investigate the impacts of two schemes: using appropriate large learning rates, and training in lower precision: EB tickets are observed to survive well under them both.

Original: [Figure]
Revised: [Figure]

Revised: generation; 2) some EB tickets are able to outperform even their unpruned, fully-trained models (e.g., the dashlines), potentially thanks to the sparse regularization learned by EB tickets.

Original: Here [0, 100] represents[0 LR
Revised: Here [0, 100] represents

Original: LR LR Schedule Retrain acc. ( % ) (CIFAR-100)
Revised: [Table]

Original: Retrain acc.
Revised: acc.

Original: ( % ) (CIFAR-10)
Revised: [Table] 10)   ﬁnal

Original:   ﬁnalﬁnal
Revised:   ﬁnal

Original: VGG [0, 100] 66.
Revised: VGG [0, 100] 66.

Original: PreResNet [0, 100] 69.
Revised: PreResNet [0, 100] 69.

Revised: 3. D O E ARLY - BIRD T ICKETS S TILL E MERGE UNDER L OWER -C OST T RAINING ?

Revised: For EB tickets, only the important connections found by them (connectivity) matter, while the weights are to be re-trained anyway.

Revised: One might hence come up with an idea, that more aggressive and “cheaper” training methods might be applicable to further shrink the cost of ﬁnding EB tickets (which already implies early stopping of training), as long as the same signiﬁcant connections still emerge.

Revised: We experimentally investigate the impacts of two schemes: using appropriate large learning rates, and training in lower precision: EB tickets are observed to survive well under them both.

Original: Appropriate large learning rates are important the emergence of EB tickets.
Revised: Appropriate large learning rates are important the emergence of EB tickets.

Original: We ﬁrst vary the learning rate schedule in the above experiments.
Revised: We ﬁrst vary the learning rate schedule in the above experiments.

Original: The original schedule is denoted as [80 LR → 0 . 01 , 120 LR → 0 . 001 ] , i.e., starting at 0.1, decayed to 0.01 at epoch 80, and further decayed to 0.001 at epoch 120.
Revised: The original schedule is denoted as [80 LR → 0 . 01 , 120 LR → 0 . 001 ] , i.e., starting at 0.1, decayed to 0.01 at epoch 80, and further decayed to 0.001 at epoch 120.

Original: In comparison, we test a new learning rate schedule [0 LR → 0 . 01 , 100 LR → 0 . 001 ] : starting at 0.01 (epoch 0), and decayed to 0.001 at epoch 100.
Revised: In comparison, we test a new learning rate schedule [0 LR → 0 . 01 , 100 LR → 0 . 001 ] : starting at 0.01 (epoch 0), and decayed to 0.001 at epoch 100.

Original: After drawing tickets, we re-train them all using the same learning rate schedule [80 LR → 0 . 01 , 120 LR → 0 . 001 ] for sufﬁcient training and fair comparison.
Revised: After drawing tickets, we re-train them all using the same learning rate schedule [80 LR → 0 . 01 , 120 LR → 0 . 001 ] for sufﬁcient training and fair comparison.

Original: We see from Table 1{+Tab. 
+} that high-quality EB tickets always emerge earlier when searching with the larger [80 LR → 0 . 01 , 120 LR → 0 . 001 ] schedule, whose ﬁnal accuracies are also better.
Revised: We see from {+Tab. 
+} that high-quality EB tickets always emerge earlier when searching with the larger [80 LR → 0 . 01 , 120 LR → 0 . 001 ] schedule, whose ﬁnal accuracies are also better.

Original: Note that the earlier emergence of good EB tickets contributes to lowering the training costs.
Revised: Note that the earlier emergence of good EB tickets contributes to lowering the training costs.

Original: It has already been observed that larger learning rates are beneﬁcial for training the dense model fully to draw the winning tickets (Frankle & Carbin, 2019; Liu et al., 2018): we show this beneﬁt extends to EB tickets too.
Revised: It has already been observed that larger learning rates are beneﬁcial for training the dense model fully to draw the winning tickets (Frankle & Carbin, 2019; Liu et al., 2018): we show this beneﬁt extends to EB tickets too.

Original: More experiments with larger learning rates can be found in Appendix A.2.
Revised: More experiments with larger learning rates can be found in Appendix A.2.

Original: Low-precision training does not destroy EB tickets.
Revised: Low-precision training does not destroy EB tickets.

Original: We next examine the emergence of EB tickets within a state-of-the-art low-precision training algorithm (Wu et al., 2018b) , where both model weights, activations, gradients and errors are quantized to 8 bits throughout training.
Revised: We next examine the emergence of EB tickets within a state-of-the-art low-precision training algorithm (Wu et al., 2018b) , where all model weights, activations, gradients and errors are quantized to 8 bits throughout training.

Original: Note that we 
Revised: Note that we only apply low-precision training to the stage of identifying EB tickets, and afterwards the tickets are trained in the same full-precision as in Section 3.1.

Original: [Figure]
Revised: Fig.

Original: EBticketWinningticket
Revised: 3 shows the retraining accuracy and total number of FLOPs (EB ticket search (8 bits) + retraining (32 bits ﬂoating points)) for VGG and CIFAR-10/100. We can see that EB tickets still emerge when using aggressively low-precision for identifying EB tickets: the general trends resemble the full-precision training cases in Fig. 1, except the emergence of good EB tickets seem to become even earlier up to initial epochs. In this way, it will lead to cost savings in ﬁnding EB tickets, since low-precision updates can aggressively save energy compared to full precision, as in the Tab. 2.

Original: Epoch subnetwork drawn from
Revised: Epoch subnetwork drawn from

Original: E p o c
Revised: E p o c

Original: h s u b n e t w o r k d r a w n f r o m
Revised: h s u b n e t w o r k d r a w n f r o m

Original: VGG ( p =0.2) 0.
Revised: VGG ( p =0.2) 0.

Original: VGG ( p =0.4) 0.
Revised: VGG ( p =0.4) 0.

Original: VGG ( p =0.6) 0.
Revised: VGG ( p =0.6) 0.

Original: VGG ( p =0.8) 0.
Revised: VGG ( p =0.8) 0.

Original: PreResNet ( p =0.2) 0.
Revised: PreResNet ( p =0.2) 0.

Original: PreResNet ( p =0.4) 0. PreResNet ( p =0.6) 0.
Revised: PreResNet ( p =0.4) 0. PreResNet ( p =0.6) 0.

Original: PreResNet ( p =0.8) 0.
Revised: PreResNet ( p =0.8) 0.

Original: only apply low-precision training to the stage of identifying EB tickets, and afterwards the tickets are trained in the same full-precision as in Section 3.1. Fig.
Revised: 3. H OW TO I DENTIFY E ARLY

Original: 2 shows the retraining accuracy and total number of FLOPs (EB ticket search (8 bits) + retraining (32 bits ﬂoating points)) for VGG and CIFAR-10/100. We can see that EB tickets still emerge when using aggressively low-precision for identifying EB tickets: the general trends resemble the full-precision training cases in Fig. 1, except the emergence of good EB tickets seem to become even earlier up to initial epochs. In this way, it will lead to cost savings in ﬁnding EB tickets, since low-precision updates can aggressively save energy compared to full precision, as in the Tab.
Revised: [Figure]

Original: 3. H OW TO I DENTIFY E ARLY -B IRD T ICKETS P RACTICALLY ?
Revised: B IRD T ICKETS P RACTICALLY ?

Original: Distance between Ticket Masks.
Revised: Distance between Ticket Masks.

Original: For each time of pruning, we deﬁne a binary mask of the drawn ticket (pruned subnetwork) w.r.t.
Revised: For each time of pruning, we deﬁne a binary mask of the drawn ticket (pruned subnetwork) w.r.t.

Original: the full dense network.
Revised: the full dense network.

Original: We follow (Liu et al., 2017) to consider the scaling factor r in batch normalization (BN) layers as indicators of the corresponding channels’ signiﬁcance.
Revised: We follow (Liu et al., 2017) to consider the scaling factor r in batch normalization (BN) layers as indicators of the corresponding channels’ signiﬁcance.

Original: Given a target pruning ratio p , we then prune the channels with top p -percentage smallest r values.
Revised: Given a target pruning ratio p , we then prune the channels with top p -percentage smallest r values.

Original: Denote the pruned channels as 0 while the kept ones as 1, the original network can be mapped into a binary “ticket mask”.
Revised: Denote the pruned channels as 0 while the kept ones as 1, the original network can be mapped into a binary “ticket mask”.

Original: For any two sub-networks pruned from the same dense model, we calculate their mask distance via the Hamming distance between their two ticket masks.
Revised: For any two sub-networks pruned from the same dense model, we calculate their mask distance via the Hamming distance between their two ticket masks.

Original: Detecting EB Tickets via Mask Distances.
Revised: Detecting EB Tickets via Mask Distances.

Original: We ﬁrst visualize and observe the global behaviors of mask distances between consecutive epochs.
Revised: We ﬁrst visualize and observe the global behaviors of mask distances between consecutive epochs.

Original: Fig.
Revised: Fig.

Original: 3 plots the pairwise mask distance matrices ( 160 × 160 ) of the VGG16 and PreResNet101 experiments on CIFAR100 (from Section 3.1), at different pruning ratios p , where ( i, j ) -th
Revised: 2 plots the pairwise mask distance matrices ( 160 × 160 ) of the VGG16 and PreResNet101 experiments on CIFAR-100 (from Section 3.1), at different pruning ratios p , where ( i, j ) -th

Original: element in a matrix denotes the mask distance between epochs i and j in that corresponding experiment (160 epochs in total for all).
Revised: element in a matrix denotes the mask distance between epochs i and j in that corresponding experiment (160 epochs in total for all).

Original: For the ease of visualization, all elements in a matrix are linearly normalized between 0 and 1; Therefore in the resulting matrix, a lower value (close to 0) indicates a smaller mask distance and is colored warmer (same hereinafter).
Revised: For the ease of visualization, all elements in a matrix are linearly normalized between 0 and 1; Therefore in the resulting matrix, a lower value (close to 0) indicates a smaller mask distance and is colored warmer (same hereinafter).

Original: Fig.
Revised: Fig.

Original: 3 demonstrates fairly consistent behaviors.
Revised: 2 demonstrates fairly consistent behaviors.

Original: Taking VGG16 on CIFAR100 ( p = 0.2) for example: 1) in the beginning, the mask distances change rapidly between epochs, manifested by the quickly “cooling” colors (yellow → green), from the diagonal line (lowest since that is comparing an epoch’s mask with itself), to off-diagonal (comparing different epochs; the more an element deviates from the diagonal, the more distant the two epochs are away from each other); 2) after  10 epochs, the off-diagonal elements become “yellow” too, and the color transition becomes much smoother from diagonal to off-diagonal, indicating the masks change only very mildly after passing this point; 3) after  80 epochs, the mask distances almost see no change across epochs.
Revised: Taking VGG16 on CIFAR-100 ( p = 0.2) for example: 1) in the beginning, the mask distances change rapidly between epochs, manifested by the quickly “cooling” colors (yellow → green), from the diagonal line (lowest since that is comparing an epoch’s mask with itself), to off-diagonal (comparing different epochs; the more an element deviates from the diagonal, the more distant the two epochs are away from each other); 2) after ∼ 10 epochs, the off-diagonal elements become “yellow” too, and the color transition becomes much smoother from diagonal to off-diagonal, indicating the masks change only very mildly after passing this point; 3) after ∼ 80 epochs, the mask distances almost see no change across epochs.

Original: Similar trends are observed in other plots too.
Revised: Similar trends are observed in other plots too.

Original: It seems to concur the hypothesis in (Achille et al., 2019) that a network ﬁrst learns important connectivity patterns and then ﬁxs them to tune weights more. 
Revised: It seems to concur the hypothesis in (Achille et al., 2019) that a network ﬁrst learns important connectivity patterns and then ﬁxs them to tune weights more. [Figure]

Original: Our observation that the ticket masks quickly become stable and hardly changed in early training stages supports drawing EB tickets.
Revised: Our observation that the ticket masks quickly become stable and hardly changed in early training stages supports drawing EB tickets.

Original: We therefore measure the mask distance consecutive epochs, and draw EB tickets when such distance is smaller than a threshold (cid:15) .
Revised: We therefore measure the mask distance consecutive epochs, and draw EB tickets when such distance is smaller than a threshold (cid:15) .

Original: Practically, to improve the reliability of EB tickets, we will stop to draw EB tickets when the last ﬁve recorded mask distances are all smaller than (cid:15) , to avoid some irregular ﬂuctuation in early training stages.
Revised: Practically, to improve the reliability of EB tickets, we will stop to draw EB tickets when the last ﬁve recorded mask distances are all smaller than (cid:15) , to avoid some irregular ﬂuctuation in early training stages.

Original: In Fig.
Revised: In Fig.

Original: 3, the red lines indicate the identiﬁcation of EB tickets when (cid:15) is set to 0 .
Revised: 2, the red lines indicate the identiﬁcation of EB tickets when (cid:15) is set to 0 .

Original:  E FFICIENT T RAINING VIA E ARLY B IRD T ICKETS
Revised:  E FFICIENT T RAINING VIA E ARLY B IRD T ICKETS

Original: In this section, we present an efﬁcient DNN training scheme which leverages the existence of EB tickets and the proposed mask distance that can detect the emergence of EB tickets for time- and energy-efﬁcient training: we call it EB Train.
Revised: In this section, we present an efﬁcient DNN training scheme which leverages the existence of EB tickets and the proposed mask distance that can detect the emergence of EB tickets for timeand energy-efﬁcient training: we call it EB Train.

Original: We will ﬁrst provide a conceptual overview, and then describe the routine of EB Train, and ﬁnally show the evaluation performance of EB Train by benchmarking it with state-of-the-art methods of obtaining compressed DNN models based on representative datasets and DNNs.
Revised: We will ﬁrst provide a conceptual overview, and then describe the routine of EB Train, and ﬁnally show the evaluation performance of EB Train by benchmarking it with state-of

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] [Figure]
Revised: -the-art methods of obtaining compressed DNN models based on representative datasets and DNNs.

Original: [Figure]
Revised: [Equation]

Original: EB Train vs. Progressive Pruning and Training .
Revised: EB Train vs. Progressive Pruning and Training .

Original: Fig.
Revised: Fig.

Original: 4 illustrates an overview of our proposed
Revised: 4 illustrates an overview of our proposed

Original: EB Train and the progressive pruning and training scheme, e.g., as in (Frankle & Carbin, 2019).
Revised: EB Train and the progressive pruning and training scheme, e.g., as in (Frankle & Carbin, 2019).

Original: In particular, the progressive pruning and training scheme adopts a three-step routine of 1) training a large and dense model, 2) pruning it and 3) then retraining the pruned model to restore performance, and these three steps can be iterated (Han et al., 2015).
Revised: In particular, the progressive pruning and training scheme adopts a three-step routine of 1) training a large and dense model, 2) pruning it, and 3) then retraining the pruned model to restore performance, and these three steps can be iterated (Han et al., 2015).

Original: The ﬁrst step often dominates (e.g., 75% when using the PreResNet101 model and CIFAR-10 dataset) in terms of training energy and time costs as it involves training a large and dense model.
Revised: The ﬁrst step often dominates (e.g., 75% when using the PreResNet101 model and CIFAR-10 dataset) in terms of training energy and time costs as it involves training a large and dense model.

Original: The key feature of our EB Train scheme is that it replaces the aforementioned steps 1 and 2 with a lower-cost step of detecting the EB tickets, i.e., enabling early stopping during the time- and energy-dominant step of training the large and dense model, thus promising large savings in both training time and energy.
Revised: The key feature of our EB Train scheme is that it replaces the aforementioned steps 1 and 2 with a lower-cost step of detecting the EB tickets, i.e., enabling early stopping during the time- and energy-dominant step of training the large and dense model, thus promising large savings in both training time and energy.

Original: For example, assuming the ﬁrst step of the progressive pruning and training scheme requires N epochs to sufﬁciently train the model for achieving the target accuracy, the proposed EB Train needs only N EB epochs to identify the EB tickets making use of the mask distance that can detect the emergence of EB tickets, where
Revised: For example, assuming the ﬁrst step of the progressive pruning and training scheme requires N epochs to sufﬁciently train the model for achieving the target accuracy, the proposed EB Train needs only N EB epochs to identify the EB tickets making use of the mask distance that can detect the emergence of EB tickets, where

Original: N EB (cid:28) N (e.g., N EB /N =
Revised: N EB (cid:28) N (e.g., N EB /N =

Original: 5% in the experiments summarized in Fig.1 and N EB /N
Revised: 5% in the experiments summarized in Fig.1 and N EB /N

Original: = 6 .
Revised: = 6 .

Original: 25% in the experiment summarized in Table 1).
Revised: 25% in the experiment summarized in Tab. 1).

Original: end end

Original: 4. H OW TO I MPLEMENT EB T RAIN ?
Revised: 4. H OW TO I MPLEMENT EB T RAIN ?

Original: From Fig.
Revised: From Fig.

Original: 1, we can see that the EB Train scheme consists of a training (searching) step to identify
Revised: 1, we can see that the EB Train scheme consists of a training (searching) step to identify

Original: EB tickets and a ﬁne-tuning step to retrain the EB tickets for achieving the target accuracy.
Revised: EB tickets and a retraining step to retrain the EB tickets for achieving the target accuracy.

Original: As the ﬁne-tuning step is the same as that of the progressive pruning and training baseline, we here only elaborate the ﬁrst step of searching EB tickets as summarized in Algorithm 1.
Revised: As the retraining step is the same as that of the progressive pruning and training baseline, we here only elaborate the ﬁrst step of searching EB tickets as summarized in Algorithm 1.

Original: Speciﬁcally, the EB searching process 1) ﬁrst initializes the weights and scaling factors W and r ; 2) iterates the structured pruning process as in (Liu et al
Revised: Speciﬁcally, the EB searching process 1) ﬁrst initializes the weights and scaling factors W and r ; 2) iterates the structured pruning process as in (Liu et al.

Original: to calculate the mask distances between the consecutive resulting subnetworks, storing them into a ﬁrst-in-ﬁrst-out (FIFO) queue of maximum length l = 5; and 3) exits when the maximum mask distance in the FIFO is smaller than a speciﬁed threshold (cid:15) (default 0.1 with the [0,1] normalized distances).
Revised: to calculate the mask distances between the consecutive resulting subnetworks, storing them into a ﬁrst-in-ﬁrst-out (FIFO) queue of maximum length l = 5; and 3) exits when the maximum mask distance in the FIFO is smaller than a speciﬁed threshold (cid:15) (default 0.1 with the [0,1] normalized distances).

Original: The output is the resulting EB ticket denoted as f ( x ; m t ∗ (cid:12) W ) , which will be retrained further to reach the target accuracy.
Revised: The output is the resulting EB ticket denoted as f ( x ; m t ∗ (cid:12) W ) , which will be retrained further to reach the target accuracy.

Original: While low-precision ticket search has already been proven to be completely feasible and favorable (see Fig. 2), we do NOT include it into current efﬁcient training experiments.
Revised: While low-precision ticket search has already been proven to be completely feasible and favorable (see Fig. 3), we do NOT include it into current efﬁcient training experiments.

Original: That is to purposely avoid the over-complicacy that other training algorithms can also be implemented in the same low precision regimes.
Revised: That is to purposely avoid the over-complicacy that other training algorithms can also be implemented in the same low precision regimes.

Original: We do, however, note that EB Train can further aggressively shrink its computation/energy costs, without sacriﬁcing accuracy, by adopting the low-precision ticket search described in Section 3.2.
Revised: We do, however, note that EB Train can further aggressively shrink its computation/energy costs, without sacriﬁcing accuracy, by adopting the low-precision ticket search described in Section 3.2.

Original: We leave this for the promising next-step work.
Revised: We leave this for the promising next-step work.

Original: In the following experiments, both
Revised: In the following experiments, both

Original: EB Train and its competitors use 32-bit ﬂoating point implementations to ensure fairness.
Revised: EB Train and its competitors use 32-bit ﬂoating point implementations to ensure fairness.

Original: 4. H OW DOES EB T RAIN C OMPARE TO STATE - OF - THE - ART T ECHNIQUES ?
Revised: 4. H OW DOES EB T RAIN C OMPARE TO S

Original: Experiment Setting.
Revised: Experiment Setting.

Original: We consider training VGG16 and PreResNet101 models on both CIFAR10/100 and ImageNet datasets.
Revised: We consider training VGG16 and PreResNet101 models on both CIFAR10/100 and ImageNet datasets following the basic setting of (Liu et al., 2018) (i.e., trainingepochs for CIFAR and 90 epochs for ImageNet).

Original: We measure the training energy costs from real-device operations: as the energy cost consists of both computational and data movement costs, the latter of which is often dominant but can not captured by the commonly used metrics, such as the number of FLOPs (Chen et al., 2017), we evaluate the proposed techniques against the baselines in terms of accuracy and real measured energy consumption.
Revised: We measure the training energy costs from realdevice operations: as the energy cost consists of both computational and data movement costs, the latter of which is often dominant but can not captured by the commonly used metrics, such as the number of FLOPs (Chen et al., 2017), we evaluate the proposed techniques against the baselines in terms of accuracy and real measured energy consumption.

Original: Speciﬁcally, all the energy consumption of full precision models are obtained by training the corresponding models in an embedded GPU (NVIDIA JETSON TX2).
Revised: Speciﬁcally, all the energy consumption of full precision models are obtained by training the corresponding models in an embedded GPU (NVIDIA JETSON TX2).

Original: The GPU measurement setup can be found in the Appendix.
Revised: The GPU measurement setup can be found in the Appendix.

Original: Note that the energy measurement results include the overhead of using the mask distance to detect the emergence of EB tickets, which is found negligible as compared to the total training cost.
Revised: Note that the energy measurement results include the overhead of using the mask distance to detect the emergence of EB tickets, which is found negligible as compared to the total training cost.

Original: For example, for the VGG16 model, the overhead caused by computing mask distances is ≤ 0 .
Revised: For example, for the VGG16 model, the overhead caused by computing mask distances is ≤ 0 .

Original: 029% in memory storage size, ≤ 0 .
Revised: 029% in memory storage size, ≤ 0 .

Original: 026% in FLOPs, and ≤ 0 .
Revised: 026% in FLOPs, and ≤ 0 .

Original: 07% in real-measured energy costs.
Revised: 07% in real-measured energy costs.

Original: This is because 1) the mask distance evaluation only involves simple Hamming distance calculations and 2) each epoch only calculates the distance once.
Revised: This is because 1) the mask distance evaluation only involves simple Hamming distance calculations and 2) each epoch only calculates the distance once.

Original: Results and Analysis.
Revised: Results and Analysis.

Original: We ﬁrst compare the computational savings (total number of FLOPs) with the baseline using a progressive pruning and training scheme (Liu et al., 2017), when drawing subnetworks at different epochs.
Revised: We ﬁrst compare the computational savings (total number of FLOPs) with the baseline using a progressive pruning and training scheme (Liu et al., 2017), when drawing subnetworks at different epochs.

Original: Fig.
Revised: Fig.

Original: 5 summarizes the results of the PreResNet/VGG16 models and CIFAR-10/100 datasets, corresponding to the same set of experiments as in Fig.
Revised: 5 summarizes the results of the PreResNet101/VGG16 models and CIFAR-10/100 datasets, corresponding to the same set of experiments as in Fig.

Original: We see that EB Train can achieve 2 .
Revised: We see that EB Train can achieve 2 .

Original: 2 − 2 .
Revised: 2 ∼ 2 .

Original: 4 × FLOPs reduction compared to the common baseline, while leading to comparable or even better accuracy (- 0.81% ∼ + 2.38% over the baseline).
Revised: 4 × FLOPs reduction compared to the common baseline, while leading to comparable or even better accuracy (- 0.81% ∼ + 2.38% over the baseline).

Original: Table 2 compares the retraining accuracy and consumed FLOPs/energy of EB Train with four stateof-the-art progressive pruning and training techniques: the original lottery ticket (LT) training (Frankle & Carbin, 2019), network slimming (NS) (Liu et al., 2017), ThiNet (Luo et al., 2017) and SNIP(Lee et al., 2019).
Revised: Tab. 2 compares the retraining accuracy and consumed FLOPs/energy of EB Train with four stateof-the-art progressive pruning and training techniques: the original lottery ticket (LT) training (Frankle & Carbin, 2019), network slimming (NS) (Liu et al., 2017), ThiNet (Luo et al., 2017) and SNIP

Original: While all of them involve the process of training a dense network, pruning it, and retraining, they apply different pruning criteria, e.g., NS imposes L 1 -sparsity on channel-wise scaling factors from BN layers, and ThiNet greedily prunes the channel that has the smallest effect on the next layers activation values.

Original: For EB Train, we by default follow (Liu et al., 2017) to inherit the

Original: [Table]
Revised: [Table]

Original: 2038/ 1614/ 1171/59.8 EB

Original: [Table] [Table] EB [Table]
Revised: (Lee et al., 2019).

Original: Improv.

Original: [-7.0 × /1.
 × 7.2 × /1.
 × 7.6 × /2.
 × p=10% p=30% p=50% p=10% p=30% p=50%-]
Revised: [-7.0 × /1.
 × 7.2 × /1.
 × 7.6 × /2.
 × p=10% p=30% p=50% p=10% p=30% p=50%-]While all of them involve the process of training a dense network, pruning it, and retraining, they apply different pruning criteria, e.g., NS imposes L 1 -sparsity on channel-wise scaling factors from BN layers, and ThiNet greedily prunes the channel that has the smallest effect on the next layers activation values.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: NS

Original: [Table]

Original: 636.5/ 592.3/ 567.8/480 ThiNet

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table] 345.3/ 300.0/ 246.8/212 EB [Table] [Table]
Revised: For EB Train, we by default follow (Liu et al., 2017) to inherit the same weight when re-training the searched ticket.

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: 6.8 × /1.1 × 5.

Original: [Table]

Original: [-[Table] .2 × 10.7 × /1.
 × same weight when re-training the searched ticket.-] We also notice existing debates (Liu et al., 2018) on the initialization re-use, and thus also compare with a variant by re-training the ticket from a new random initialization, termed as EB Train (re-init) .
Revised: [-[Table] .2 × 10.7 × /1.
 × same weight when re-training the searched ticket.-] We also notice existing debates (Liu et al., 2018) on the initialization re-use, and thus also compare with a variant by re-training the ticket from a new random initialization, termed as EB Train (re-init) .

Original: Furthermore, we apply the proposed EB Train on top of the low precision training method in (Yang et al., 2019a) and obtain experiment results of another two variants of EB Train: 1) EB Train with low precision search and full precision retrain (i.e., EB Train LF), and 2) EB Train with both low precision search and retrain (i.e., EB Train LL).
Revised: Furthermore, we apply the proposed EB Train on top of the low precision training method in (Yang et al., 2019a) and obtain experiment results of another two variants of EB Train: 1) EB Train with low precision search and full precision retrain (i.e., EB Train LF), and 2) EB Train with both low precision search and retrain (i.e., EB Train LL).

Original: Table 2 demonstrates that EB Train consistently outperforms all competitors in terms of saving training energy and computation costs, meanwhile improving the ﬁnal accuracy in most cases.
Revised: Tab. 2 demonstrates that EB Train consistently outperforms all competitors in terms of saving training energy and computation costs, meanwhile improving the ﬁnal accuracy in most cases.

Original: We use EB
Revised: We use EB

Original: Train Improv.
Revised: Train Improv.

Original: to record the performance margin (either accuracy, or energy/computation) between
Revised: to record the performance margin (either accuracy, or energy/computation) between

Original: EB Train, and the strongest competitor among the four state-of-the-art baselines.
Revised: EB Train, and the strongest competitor among the four state-of-the-art baselines.

Original: Speciﬁcally, EB
Revised: Speciﬁcally, EB

Original: Train with full precision ﬂoating point (FP32) search and retrain outperforms those pruning methods by up to 1 .
Revised: Train with full precision ﬂoating point (FP32) search and retrain outperforms those pruning methods by up to 1 .

Original: 2 − 4 .
Revised: 2 ∼ 4 .

Original: 7 × and 1 .
Revised: 7 × and 1 .

Original: 1 − 4 .
Revised: 1 ∼ 4 .

Original: 5 × in terms of the energy consumption and computational
Revised: 5 × in terms of the energy consumption and computational

Original: FLOPs, while always achieving comparable or even better accuracies, across three pruning ratios, two DNN models and two datasets.
Revised: FLOPs, while always achieving comparable or even better accuracies, across three pruning ratios, two DNN models and two datasets.

Original: Moreover, EB Train with both low precision (8-bits block ﬂoating point (FP8)) search and retrain outperforms the baselines by up to 5 .
Revised: Moreover, EB Train with both low precision (8-bits block ﬂoat- [Table] ing point (FP8)) search and retrain outperforms the baselines by up to 5 .

Original: 8 − 24 .
Revised: 8 ∼ 24 .

Original: 6 × and 1 .
Revised: 6 × and 1 .

Original: 1 − 5 . [-
-] × in terms of energy consumption estimated using real-measured unit energy from (Yang et al., 2019b) and computational FLOPs.
Revised: 1 ∼ 5 . [-
-]0 × in terms of energy consumption estimated using real-measured unit energy from (Yang et al., 2019b) and computational FLOPs.

Original: In addition, comparing with the re-init variant endorses the effectiveness[Table] of initialization inheritance in EB Train.
Revised: In addition, comparing with the re-init variant endorses the effectiveness of initialization inheritance in EB Train.

Original: As an additional highlight, EB Train naturally leads to pruned DNN models whose inferences are efﬁcient too, unifying the boost of efﬁciency throughout the full learning lifecycle.
Revised: As an additional highlight, EB Train naturally leads to pruned DNN models whose inferences are efﬁcient too, unifying the boost of efﬁciency throughout the full learning lifecycle.

Original: ResNet18 on ImageNet.
Revised: ResNet18/50 on ImageNet.

Original: To study the performance of EB Train in a harder dataset, we conduct experiments using ResNet18 and ImageNet, and compare its resulting accuracy and total training FLOPs with those of the method in (He et al., 2016a), NS (Liu et al.,  2017) and SFP (He et al., 2018) as summarized in Table 3.
Revised: To study the performance of EB Train in a harder dataset, we conduct experiments using ResNet18/50 and ImageNet, and compare its resulting accuracy and total training FLOPs with those of the method in (He et al., 2016a), NS (Liu et al., 2017), ThiNet (Luo et al., 2017) and SFP (He et al., 2018) as summarized in Tab. 3.

Original: We can see that 1) when the pruning ratio is 10%, EB Train achieves a better accuracy (+0.27 vs. +0.08 over the unpruned one) while requiring 51% less total training FLOPs, as compared to the NS pruning method; 2)when the pruning ratio is 30%, EB 
Revised: We have three observations : 1) When training ResNet18 on ImageNet, EB Train achieves a better accuracy (+0.27%) over the unpruned one while introducing 10% channel sparsity; 2) EB Train outperforms other baselines for both ResNet18 and ResNet50 by reducing the total training FLOPs while achieving a comparable or better accuracy.

Original:  Train results in a higher{+better top-1 accuracy of +0.
 % ∼ +1.18% and +1.74% ∼ +2.34% for ResNet18 and ResNet50, respectively. For example, when training ResNet50 on ImageNet, EB Train with a pruning ratio of 70% achieves better (+1.74%)+} accuracy(-1.29 vs. -1.72(NS)/-2.47(SFP) over the unpruned one) while reducing the training FLOPs by 56% and 52%, as compared to the NS and SFP baselines.
Revised: Speciﬁcally, EB Train achieves a reduced training FLOPs of 51.5% ∼ 56.1% and 48.6% ∼ 74.0% with a {+better top-1 accuracy of +0.
 % ∼ +1.18% and +1.74% ∼ +2.34% for ResNet18 and ResNet50, respectively. For example, when training ResNet50 on ImageNet, EB Train with a pruning ratio of 70% achieves better (+1.74%)+} accuracy over ThiNet while saving 74% total training FLOPs; 3) Different from results on CIFAR shown in Tab. 2, all methods performed on ImageNet (harder dataset) start to yield accuracy reductions (-1.72% ∼ -3.55%) for ResNet18/ of only 30% pruning ratio, as compared to unpruned one.

Original: Note that in Table 3, the unpruned results are based on the ofﬁcial implementation in (He et al., 2016a); The SFP results are obtained from their original paper (
Revised: Note that in Tab. 3, the unpruned results are based on the ofﬁcial implementation in (He et al., 2016a); The SFP results are obtained from their original paper (

Original: He et al., 2018), which does not provide results at a 10% pruning ratio;  and the NS results are obtained by conducting the experiments ourselves.
Revised: He et al., 2018), which does not provide results at a 10% pruning ratio; all the ThiNet results at different pruning ratios are obtained from original paper (Luo et al., 2017); and the NS results are obtained by conducting the experiments ourselves.

Original:  D ISCUSSION AND CONCLUSION
Revised:  D ISCUSSION AND CONCLUSION

Original: We have demonstrated that winning tickets can be drawn at the very early training stage, i.e., EB tickets exist, in both the standard training and several lower-cost training schemes.
Revised: We have demonstrated that winning tickets can be drawn at the very early training stage, i.e., EB tickets exist, in both the standard training and several lower-cost training schemes.

Original: That motivates a practical success of applying EB tickets to efﬁcient training, whose results compare favorably against state-of-the-arts.
Revised: That motivates a practical success of applying EB tickets to efﬁcient training, whose results compare favorably against state-of-the-arts.

Original: Moreover, experiments show that EB Train with low-precision search and retrain achieve more efﬁcient training.
Revised: Moreover, experiments show that EB Train with low-precision search and retraining achieve more efﬁcient training.

Original: We believe many promising problems still remain open to be addressed.
Revised: We believe many promising problems still remain open to be addressed.

Original: An immediate future work is to test low-precision EB Train algorithms on larger models/datasets.
Revised: An immediate future work is to test low-precision EB Train algorithms on larger models/datasets.

Original: We are also curious whether more lower-cost training techniques could be associated with EB tickets.
Revised: We are also curious whether more lower-cost training techniques could be associated with EB Train.

Original: Finally, sometimes high pruning ratios (e.g, p ≥ 0.7) can hurt the quality of EB tickets and the retrained networks.
Revised: Finally, sometimes high pruning ratios (e.g, p ≥ 0.7) can hurt the quality of EB tickets and the retrained networks.

Original: We look forward to automating the choice of p for different models/datasets, unleashing higher efﬁciency.
Revised: We look forward to automating the choice of p for different models/datasets, unleashing higher efﬁciency.

Original: A PPROPRIATE LARGE LEARNING RATE FAVOR THE EMERGENCE OF EB TICKETS
Revised: A PPROPRIATE LARGE LEARNING RATE FAVOR THE EMERGENCE OF EB TICKETS

Original: Here we show the effect of larger learning rates setting.
Revised: Here we show the effect of larger learning rates setting.

Original: The original schedule is denoted as [80 LR → 0 . 01 , 120 LR → 0 . 001 ] , i.e., starting at 0.1, decayed to 0.01 at epoch 80, and further decayed to 0.001 at epoch 120.
Revised: The original schedule is denoted as [80 LR → 0 . 01 , 120 LR → 0 . 001 ] , i.e., starting at 0.1, decayed to 0.01 at epoch 80, and further decayed to 0.001 at epoch 120.

Original: In comparison, we test a larger initial learning rate, staring at 0.5 for VGG and starting at 0.2 for PreResNet101.
Revised: In comparison, we test a larger initial learning rate, staring at 0.5 for VGG and starting at 0.2 for PreResNet101.

Original: After drawing tickets, we re-train them all using the same learning rate schedule with their searching phase for sufﬁcient training and fair comparison.
Revised: After drawing tickets, we re-train them all using the same learning rate schedule with their searching phase for sufﬁcient training and fair comparison.

Original: We see from Table 4{+Tab. 
+} that high-quality EB tickets also emerge early when searching with the larger initial learning rate, whose ﬁnal accuracies are even better.
Revised: We see from {+Tab. 
+} that high-quality EB tickets also emerge early when searching with the larger initial learning rate, whose ﬁnal accuracies are even better.

Original: [Table]
Revised: [Table]

Original: PreResNet 0.
Revised: PreResNet 0.