CPT: E FFICIENT D EEP N EURAL N ETWORK T RAINING VIA C YCLIC P RECISION
A BSTRACT
Low-precision deep neural network (DNN) training has gained tremendous attention as reducing precision is one of the most effective knobs for boosting DNNs’ training time/energy efﬁciency.
In this paper, we attempt to explore low-precision training from a new perspective as inspired by recent ﬁndings in understanding
DNN training: we conjecture that DNNs’ precision might have a similar effect as the learning rate during DNN training, and advocate dynamic precision along the training trajectory for further boosting the time/energy efﬁciency of DNN training.
Speciﬁcally, we propose Cyclic Precision Training (CPT) to cyclically vary the precision between two boundary values which can be identiﬁed using a simple precision range test within the ﬁrst few training epochs.
Extensive simulations and ablation studies on ﬁve datasets and ten models demonstrate that CPT’s effectiveness is consistent across various models/tasks (including classiﬁcation and language modeling).
Furthermore, through experiments and visualization we show that CPT helps to (1) converge to a wider minima with a lower generalization error and (2) reduce training variance which we believe opens up a new design knob for simultaneously improving the optimization and efﬁciency of DNN training.
Our codes are available at: https://github.com/RICE-EIC/CPT.
I NTRODUCTION
The record-breaking performance of modern deep neural networks (DNNs) comes at a prohibitive training cost due to the required massive training data and parameters, limiting the development of the highly demanded DNN-powered intelligent solutions for numerous applications (Liu et al., 2018; Wu et al., 2018).
As an illustration, training ResNet-50 involves 10 18 FLOPs (ﬂoating-point operations) and can take 14 days on one state-of-the-art (SOTA)
GPU (You et al., 2020b).
Meanwhile, the large DNN training costs have raised increasing ﬁnancial and environmental concerns.
For example, it is estimated that training one DNN can cost more than $10K US dollars and emit carbon as high as a car’s lifetime emissions.
In parallel, recent DNN advances have fueled a tremendous need for intelligent edge devices, many of which require on-device in-situ learning to ensure the accuracy under dynamic real-world environments, where there is a mismatch between the devices’ limited resources and the prohibitive training costs (Wang et al., 2019b; Li et al., 2020; You et al., 2020a).
To address the aforementioned challenges, extensive research efforts have been devoted to developing efﬁcient DNN training techniques.
Among them, low-precision training has gained signiﬁcant attention as it can largely boost the training time/energy efﬁciency (Jacob et al., 2018;
Wang et al., 2018a; Sun et al., 2019).
For instance, GPUs can now perform mixed-precision DNN training with 16-bit IEEE Half-Precision ﬂoating-point formats (Micikevicius et al., 2017b).
Despite their promise, existing low-precision works have not yet fully explored the opportunity of leveraging recent ﬁndings in understanding DNN training.
In particular, existing works mostly ﬁx the model precision during the whole training process, i.e., adopt a static quantization strategy, while recent works in DNN training optimization suggest dynamic hyper-parameters along DNNs’ training trajectory.
For example, (Li et al., 2019) shows that a large initial learning rate helps the model to memorize easier-to-ﬁt and more generalizable patterns, which aligns with the common practice to start from a large learning rate for exploration and anneal to a small one for ﬁnal convergence; and (Smith, 2017; Loshchilov & Hutter, 2016) improve DNNs’ classiﬁcation accuracy by adopting cyclical learning rates.
In this work, we advocate dynamic precision training, and make the following contributions:
• We show that DNNs’ precision seems to have a similar effect as the learning rate during DNN training,
i
e., low precision with large quantization noise helps DNN training exploration while high precision with more accurate updates aids model convergence, and dynamic precision schedules help DNNs converge to a better minima.
This ﬁnding opens up a design knob for simultaneously improving the optimization and efﬁciency of DNN training.
• We propose Cyclic Precision Training (CPT) which adopts a cyclic precision schedule along DNNs’ training trajectory for pushing forward the achievable trade-offs between DNNs’ accuracy and training efﬁciency.
Furthermore, we show that the cyclic precision bounds can be automatically identiﬁed at the very early stage of training using a simple precision range test, which has a negligible computational overhead.
• Extensive experiments on ﬁve datasets and ten models across a wide spectrum of applications (including classiﬁcation and language modeling) validate the consistent effectiveness of the proposed CPT technique in boosting the training efﬁciency while leading to a comparable or even better accuracy.
Furthermore, we provide loss surface visualization for better understanding CPT’s effectiveness and discuss its connection with recent ﬁndings in understanding DNNs’ training optimization.
R ELATED WORKS
Quantized DNNs.
DNN quantization (Courbariaux et al., 2015; 2016; Rastegari et al., 2016; Zhu et al., 2016;
Li et al., 2016; Jacob et al., 2018; Mishra & Marr, 2017;
Mishra et al., 2017; Park et al., 2017; Zhou et al., 2016) has been well explored based on the target accuracy-efﬁciency tradeoffs.
For example, (Jacob et al., 2018) proposes quantization-aware training to preserve the post quantization accuracy; (Jung et al., 2019; Bhalgat et al., 2020; Esser et al., 2019; Park & Yoo, 2020) strive to improve low-precision DNNs’ accuracy using learnable quantizers.
Mixed-precision DNN quantization (Wang et al., 2019a; Xu et al., 2018; Elthakeb et al., 2020; Zhou et al., 2017) assigns different bitwidths for different layers/ﬁlters.
While these works all adopt a static quantization strategy, i.e., the assigned precision is ﬁxed post quantization, CPT adopts a dynamic precision schedule during the training process.
Low-precision DNN training.
Pioneering works (Wang et al., 2018a;
Banner et al., 2018; Micikevicius et al., 2017a; Gupta et al., 2015; Sun et al., 2019) have shown that DNNs can be trained with reduced precision.
For distributed learning, (Seide et al., 2014; De Sa et al., 2017; Wen et al., 2017;
Bernstein et al., 2018) quantize the gradients to reduce the communication costs, where the training computations still adopt full precision; For centralized/on-device learning, the weights, activations, gradients, and errors involved in both the forward and backward computations all adopt reduced precision.
Our CPT can be applied on top of these low-precision training techniques, all of which adopt a static precision during the whole training trajectory, to further boost the training efﬁciency.
Dynamic-precision DNNs.
There exist some dynamic precision works which aim to derive a quantized DNN for inference after the full-precision training.
Speciﬁcally, (Zhuang et al., 2018)
ﬁrst trains a full-precision model to reach convergence and then gradually decreases the model precision to the target one for achieving better inference accuracy; (Khoram & Li, 2018) also starts from a full-precision model and then gradually learns the precision of each layer to derive a mixed-precision counterpart; (Yang & Jin, 2020) learns a fractional precision of each layer/ﬁlter based on the linear interpolation of two consecutive bitwidths which doubles the computation and requires an extra ﬁne-tuning step; and (Shen et al., 2020) proposes to adapt the precision of each layer during inference in an input-dependent manner to balance computational cost and accuracy.
T HE P ROPOSED CPT T ECHNIQUE
In this section, we ﬁrst introduce the hypothesis that motivates us to develop CPT using visualization examples in Sec. 3.1, and then present the CPT concept in Sec. 3.2 followed by the Precision Range
Test (PRT) method in Sec.
3.3, where PRT aims to automate the precision schedule for CPT. [Table]
3. CPT: M OTIVATION
Hypothesis 1: DNN’s precision has a similar effect as the learning rate.
Existing works (Grandvalet et al., 1997; Neelakantan et al., 2015) show that noise can help DNN training theoretically or empirically, motivating us to rethink the role of quantization in DNN training.
We conjecture that low precision with large quantization noise helps DNN training exploration with an effect similar to a high learning rate
[Table]
while high precision with more accurate updates aids model convergence, similar to a low learning rate.
Validating Hypothesis 1.
Settings: To empirically justify our hypothesis [Table] we train ResNet-38/ on the CIFAR-100 dataset for 160 epochs following the basic training setting as in Sec. 4.1. In particular, we divide the training of 160 epochs into three stages: [0-th, 80-th], [80-th,120-th]
[Table]
and [120-th, 160-th]: for the ﬁrst training stage of [0-th, 80-th], we adopt different learning rates and precisions for the weights and activations, while using full precision for the remaining two stages with a learning rate of 0.01 for the [80-th,120-th] epochs and 0.001 for the [120-th, 160-th] epochs in all the experiments in order to explore the relationship between the learning rate and precision in the ﬁrst training stage. Results: As shown in Tab. 1, we can observe that as the learning rate is sufﬁciently reduced for the ﬁrst training stage, adopting a lower precision for this stage will lead to a higher accuracy than training with full precision. In particular, with the standard initial learning rate of 0.1, full precision training achieves a 1.00%/0.70% higher accuracy than the 4-bit one on ResNet-38/74, respectively; whereas as the initial learning rate decreases, this accuracy gap gradually narrows and then reverses, e.g., when the initial learning rate becomes 1e-2, training with [0-th, 80-th] of 4-bit achieves a 1.40%/1.57% higher accuracy than the full precision ones. Insights: This set of experiments show that (1) when the initial learning rate is low, training with lower initial precisions consistently leads to a better accuracy than training with full precision, indicating that lowering the precision introduces a similar effect of favoring exploration as that of a high learning rate; and (2) although a low precision can alleviate the accuracy drop caused by a low learning rate, a high learning rate is in general necessary to maximize the accuracy.
[Figure]
Hypothesis 2: Dynamic precision helps DNN generalization. Recent ﬁndings in DNN training have motivated us to better utilize DNN precision to achieve a win-win in both DNN accuracy and efﬁciency. Speciﬁcally, it has been discussed that (1) DNNs learn to ﬁt different patterns at different training stages, e.g., (Rahaman et al., 2019; Xu et al., 2019) reveal that DNN training ﬁrst learns lowerfrequency components and then high-frequency features, with the former being more robust to perturbations and noises; and (2) dynamic learning rate schedules help to improve the optimization in DNN training, e.g., (Li et al., 2019) points out that a large initial learning rate helps the model to memorize easier-to-ﬁt and more generalizable patterns while (Smith, 2017; Loshchilov & Hutter, 2016) show that cyclical learning rate schedules improve DNNs’ classiﬁcation accuracy.
These works inspire us to hypothesize that dynamic precision might help DNNs to reach a better optimum in the optimization landscape, especially considering the similar effect between the learning rate and precision validated in our Hypothesis 1.
[Figure] Validating Hypothesis 2.
Our Hypothesis 2 has been consistently conﬁrmed by various empirical observations.
For example, a recent work (Fu et al., 2020) proposes to progressively increase the precision during the training process, and we follow their settings to validate our hypothesis.
Settings: We train a ResNet-74 on CIFAR-100 using the same training setting as (Wang et al., 2018b) except that we quantize the weights, activations, and gradients during training; for the progressive precision case we uniformly increase the precision of weights and activations from 3-bit to 8-bit in the ﬁrst 80 epochs and adopt static 8-bit gradients, while the static precision baseline uses 8-bit for all the weights/activations/gradients. Results: Fig.
1 shows that training with progressive precision schedule achieves a slightly higher accuracy (+0.3%) than its static counterpart, while the former can reduce training costs.
Furthermore, we visualize the loss landscape (following the method in (Li et al., 2018)) in Fig.
2(b): interestingly the progressive precision schedule helps to converge to a better local minima with wider contours, indicating a lower generalization error (Li et al., 2018) over the static 8-bit baseline in Fig. 2(a). The progressive precision schedule in (Fu et al., 2020) relies on manual hyper-parameter tuning.
As such, a natural following question would be: what kind of dynamic schedules would be effective while being simple to implement for different tasks/models?
In this work, we show that a simple cyclic schedule consistently beneﬁts the training convergence while boosting the training efﬁciency.
3. CPT: T HE K EY C ONCEPT
[Figure]
(CPT), where CPT cyclically schedules the precision of weights and activations during training.
The key concept of CPT draws inspiration from (Li et al., 2019) which demonstrates that a large initial learning rate helps the model to learn more generalizable patterns.
We thus hypothesize that a lower precision that leads to a shortterm poor accuracy might actually help the DNN exploration during training thanks to its associated larger quantization noise, while it is well known that a higher precision enables the learning of higher-complexity, ﬁne-grained patterns that is critical to better convergence.
Together, this combination could improve the achieved accuracy as it might better balance coarse-grained exploration and ﬁne-grained optimization during
DNN training, which leads to the idea of CPT.
Speciﬁcally, as shown in Fig.
3, CPT varies the precision cyclically between two bounds instead of ﬁxing the precision during training, letting the models explore the optimization landscape with different granularities.
While CPT can be implemented using different cyclic scheduling methods, here we present as an example an implementation of CPT in a cosine manner: [Equation]
[Equation]
where B nmin and B nmax are the lower and upper precision bound, respectively, in the n -th cycle of precision schedule, ⌈ ⋅ ⌋ and % denote the rounding operation and the remainder operation, respectively, and B nt is the precision at the t -th global epoch which falls into the n -th cycle with a cycle length of
T n . Note that the cycle length T n is equal to the total number of training epochs divided by the total number of cycles denoted as N , where N is a hyper-parameter of CPT.
For example, if N = 2 , then a DNN training with CPT will experience two cycles of cyclic precision schedule during training.
As shown in Sec.
4.3, we ﬁnd that the beneﬁts of CPT are maintained when adopting different total number of cyclic precision schedule cycles during training, i.e., CPT is not sensitive to N .
A visualization example for the precision schedule can be found in Appendix A. Additionally, we ﬁnd that CPT is generally effective when using different dynamic precision schedule patterns (i.e., not necessarily the cosine schedule in Eq.
We implement CPT following Eq.
(1) in this work and discuss the potential variants in Sec. 4.3.
We visualize the training curve of CPT on ResNet-74 with CIFAR-100 in Fig.
1 and ﬁnd that it achieves a 0.91% higher accuracy paired with a 36.7% reduction in the required training BitOPs (bit operations), as compared to its static ﬁxed precision counterpart.
In addition, Fig. 2 (c) visualizes the corresponding loss landscape, showing the effectiveness of CPT, i.e., such a simple and automated precision schedule leads to a better convergence with lower sharpness.
[Figure]
[Figure]
The concept of CPT is simple enough to be plugged into any model or task to boost the training efﬁciency.
One remaining question is how to determine the precision bounds, i.e., B imin and B imax in Eq.
(1), which we ﬁnd can be automatically decided in the ﬁrst cycle (i.e., T
i = T 0 ) of the precision schedule using a simple PRT at a negligible computational cost.
Speciﬁcally, PRT starts from the lowest possible precision, e.g., 2-bit, and gradually increases the precision while monitoring the difference in the training accuracy magnitude averaged over several consecutive iterations; once this training accuracy difference is larger than a preset threshold, indicating that the training can at least partially converge, PRT would claim that the lower bound is identiﬁed.
While the upper bound can be similarly determined, there exists an alternative which suggests simply adopting the precision of CPT’s static precision counterpart.
The remaining cycles use the same precision bounds.
Fig.
4 visualizes the PRT for ResNet-152/MobileNetV2 trained on CIFAR-100.
We can see that the lower precision bound identiﬁed when the model experiences a notable training accuracy improvement for ResNet-152 is 3-bit while that for MobileNetV2 is 4-bit, aligning with the common observation that ResNet-152 is more robust to quantization than the more compact model MobileNetV2.
E XPERIMENT R ESULTS
In this section, we will ﬁrst describe the experiment setup in Sec.
4.1, benchmarking results over
SOTA training methods across various tasks in Sec.
4.2, and then comprehensive ablation studies of
CPT in Sec. 4.3.
4. E XPERIMENT SETUP
Models, datasets and baselines.
We consider ten models (including
ResNet18/34/38/74/110/152/164 (He et al., 2016), MobileNetV2 (Sandler et al., 2018), Transformer (Vaswani et al., 2017), and LSTM (Hochreiter & Schmidhuber, 1997)) and ﬁve tasks (including CIFAR10/100 (Krizhevsky et al., 2009), ImageNet (Deng et al., 2009), WikiText-103 (Merity et al., 2016), and Penn Treebank (PTB) (Marcus et al., 1993)).
Speciﬁcally, we follow (Wang et al., 2019b) for implementing MobileNetV2 on CIFAR-10/100.
Baselines: We ﬁrst benchmark CPT over three SOTA static low-precision training techniques: SBM (Banner et al., 2018), DoReFa (Zhou et al., 2016), and WAGEUBN (Yang et al., 2020), each of which adopts a different quantizer.
Since SBM is the most competitive baseline among the three based on both their reported and our experiment results, we apply CPT on top of SBM, and all the static precision baselines adopt the SBM quantizer unless speciﬁcally stated.
Another baseline is the cyclic learning rate (CLR) (Loshchilov & Hutter, 2016) on top of static precision training, and we follow the best setting in (Loshchilov & Hutter, 2016).
Training settings.
We follow the standard training setting in all the experiments.
In particular, for classiﬁcation tasks, we follow SOTA settings in (Wang et al., 2018b) for CIFAR-10/100 and (He et al., 2016) for ImageNet experiments, respectively; and for language modeling tasks, we follow (Baevski & Auli, 2018) for Transformer on WikiText-103 and (Merity et al., 2017) for LSTM on PTB.
Precision settings.
The lower precision bounds in all the experiments are set using the PRT in Sec.3.
and the upper bound is the same as the precision of the corresponding static precision baselines.
We only apply CPT to the weights and activations (together annotated as FW) and use static precision for the errors and gradients (together annotated as BW), the latter of which is to ensure the stability of the gradients (Wang et al., 2018a)
(more discussion in Appendix C).
In particular, CPT from 3-bit to 8-bit with 8-bit gradient is annotated as FW(3,8)/BW8.
The total number of periodic precision cycles, i.e., N in Sec.3.3, for all the experiments is ﬁxed to be 32 (see the ablation studies in Sec. 4.3).
Hardware settings and metrics.
To validate the real hardware efﬁciency of the proposed CPT, we adopt standard FPGA implementation ﬂows.
Speciﬁcally, we employ the Vivado HLx design ﬂow to implement FPGA-based accelerators on a Xilinx development board called ZC706 (Xilinx).
To better evaluate the training cost, we consider both calculated GBitOPs (Giga bit operations) and real-measured latency on the ZC706 FPGA board.
4. B ENCHMARK WITH SOTA STATIC PRECISION TRAINING METHODS
Benchmark on CIFAR-10/100.
Benchmark over SOTA quantizers:
We benchmark CPT with three
SOTA static low-precision training methods, as summarized in Tab. 2, when training ResNet-74/ and MobileNetV2 on CIFAR-10/100, covering both deep and compact DNNs which are representative difﬁcult cases of low-precision DNN training.
Note that the accuracy improvement is the difference between CPT and the strongest baseline under the same setting .
Tab. 2 shows that (1) our CPT consistently achieves a win-win with both a higher accuracy (+0.19% ∼ +1.25%) and a lower training cost (-21.0% ∼ -37.1% computational cost and -14.7% ∼ -21.4% latency) under all the cases on CIFAR-10/100, even for the compact model MobileNetV2, and (2) CPT notably outperforms the baselines in terms of accuracy under extremely low precision , which is one of the most useful scenarios for low-precision DNN training.
In particular, CPT with periodic precision between 4-bit and 6-bit boosts the accuracy by +1.25% and +1.07% on ResNet-74/164, respectively, as compared to the static 6-bit training on CIFAR-10, verifying that CPT leads to a better convergence. [Table] [Figure] [Figure]
[Table] Benchmark over CLR on top of SBM: We further benchmark CPT with CLR (Loshchilov & Hutter, 2016) (inherit its besting setting on CIFAR-100) [Table] with both being applied on top of SBM as it achieves the best performance among SOTA quantizers as shown in Tab. 2, based on more DNN models and precision as shown in Fig. 5.
We can see that (1) CPT still consistently outperforms all the baselines with a better accuracy and efﬁciency trade-off, and (2)
CLR on top of SBM leads to a negative effect on the training accuracy, which we conjecture is caused by both the instability of gradients and the sensitivity of gradients to the learning rate under low-precision training, as discussed in (Wang et al., 2018a), showing that CPT is more applicable to low-precision training than CLR.
Loss landscape visualization: To better understand the superior performance achieved by CPT, we visualize the loss landscape following the method in (Li et al., 2018), as shown in Fig.
6 covering both non-compact and compact models and two low-precision settings (i.e., 6-bit and 8-bit) which are bottlenecks for low-precision DNN training.
We can observe that, again, both standard and compact DNNs trained with CPT experience wider contours with less sharpness, indicating that CPT helps DNN training optimization to converge to better local optima.
Benchmark on ImageNet.
To verify the scalability of CPT on more complex tasks and larger models, we benchmark CPT with the SOTA static precision training method SBM (Banner et al., 2018) on
ResNet-18/34/50 with ImageNet, under which it is challenging for low-precision training such as 4-bit to work.
As shown in Tab. 3, we can observe that CPT still achieves a reduced computational cost (up to -30.4%) with a comparable accuracy (-0.20% ∼ +0.06%).
In particular, CPT works well
[Figure]
[Table]
on ResNet-50, leading to both a slightly higher accuracy and a better training efﬁciency, indicating the scalability of CPT with model complexity, and thus, its potential application in large scale training, in addition to on-device training scenarios.
[Table]
CPT for boosting accuracy: An important perspective of CPT is its potential to improve training optimality in addition to efﬁciency. We illustrate CPT’s advantage in improving the ﬁnal accuracy through training ResNet-18/34 on ImageNet using CPT and static full precision. As shown in Tab. 4, CPT on ResNet-18/34 achieves a 0.91%/0.84% higher accuracy than their full precision counterparts on ImageNet, indicating that CPT can be adopted as a general technique to improve the ﬁnal accuracy in addition to efﬁcient training.
Benchmark on WikiText-103 and PTB.
We also apply CPT on language modeling tasks (including
WikiText-103 and PTB) (see Tab. 5) to show that CPT is also applicable to natural language processing models.
Tab. 5 shows that (1) CPT again consistently achieves a win-win in terms of accuracy (i.e., perplexity - the lower the better) and training efﬁciency, and (2) language modeling models/tasks are more sensitive to quantization, especially in LSTM models, as it always adapts to a larger lower precision bound, which is consistent with SOTA observations (Hou et al., 2019).
[Table]
{+-22.4% 2-LSTM PTB SBM 
 4.03e 6 /97. 2.77e
+}
Proposed CPT 5 - 8 /96. 3.09e 5 - 6 /97. 2.48e
Improv.
4. A BLATION S TUDIES OF CPT
[Figure]
CPT with different precision ranges.
We also evaluate
CPT under a wide range of upper precision bounds, which correspond to a different target efﬁciency, to see if CPT still works well.
Fig.
7 plots a boxplot with each experiment being repeated ten times. We can see that (1) regardless of the adopted precision ranges, CPT consistently achieves a win-win (a +0.74% ∼
+2.03% higher accuracy and a -18.3% ∼ -48.0% reduction in computational cost), especially under lower precision scenarios, and (2) CPT even shrinks the accuracy variance , which better aligns with the practical goal of efﬁcient training.
CPT with different adopted numbers of precision schedule cycles.
To evaluate CPT’s sensitivity to the total number of adopted cyclic precision schedule cycles, we apply CPT on ResNet-38/74 and CIFAR-100 when using different numbers of schedule cycles. Fig.
8 plots the mean of the achieved [Figure] accuracy based on ten repeated experiments. We can see that (1) different choices for the total number of cycles lead to a comparable accuracy (within 0.5% accuracy) on CIFAR-10/100; and (2) CPT with different number of precision schedule cycles consistently outperforms the static baseline SBM in the achieved accuracy.
Based on this experiment, we set N = 32 for simplicity.
CPT with different cyclic precision schedule patterns.
We also evaluate CPT using other cyclic precision schedule patterns in addition to the cosine one, including a triangular schedule motivated by (Smith, 2017) and a cosine annealing schedule as the learning rate schedule in (Loshchilov &
Hutter, 2016), with all adopting 32 cyclic cycles for fairness.
Experiments in Tab. 6 show that (1) CPT with different schedule patterns is consistently effective, and (2) CPT with the other two schedule patterns even surpasses the cosine one in some cases but underperforms on compact models.
We leave how to determine the optimal cyclic pattern for a given model and task as a future work.
[Table]
[Figure]
MobileNetV
[Table]
D ISCUSSIONS ABOUT FUTURE WORK
The theoretical perspective of CPT.
There has been a growing interest in understanding and optimizing DNN training.
For example, (Li et al., 2019) shows that training DNNs with a large initial learning rate helps the model to memorize more generalizable patterns faster and better.
Recently, (Zhu et al., 2020) showed that under a convexity assumption the convergence bound of reduced-precision DNN training is determined by a linear combination of the quantization noise and learning rate.
These ﬁndings regarding DNN training seem to be consistent with the effectiveness of our CPT. The hardware support for CPT.
Recent progresses in mixed-precision DNN accelerators (Lee et al., 2019; Kim et al., 2020) with dedicated modules for supporting dynamic precisions are promising to support our CPT. We leave the optimal accelerator design for CPT as our future works.
C ONCLUSION
We hypothesize that DNNs’ precision has a similar effect as the learning rate when it comes to
DNN training, i.e., low precision with large quantization noise helps DNN training exploration while high precision with more accurate updates aids model convergence, and thus advocate that dynamic precision schedules help DNN training optimization.
We then propose the CPT framework which adopts a periodic precision schedule for low-precision DNN training in order to boost the achievable
Pareto frontier of task accuracy and training efﬁciency.
Extensive experiments and ablation studies verify that CPT can reduce computational cost during training while achieving a comparable or even better accuracy.
Our future work will strive to identify more theoretical grounds for such dynamic low-precision training. A CKNOWLEDGEMENT
The work is supported by the National Science Foundation (NSF) through the Real-Time Machine
Learning (RTML) program (Award number: 1937592).
A V ISUALIZATION OF THE PRECISION SCHEDULE IN CPT
Fig.
9 visualizes the precision schedule FW(3,8) with eight cycles for training on CIFAR-10/100. [Table] ResNet [Table]
FW(3,8) / BW 72.
FW(3,6) / BW 70.
[Figure]
B A BLATION STUDY : T HE STARTING EPOCH FOR ENABLING CPT
We conduct another ablation study to explore whether CPT is always necessary during the whole training process. We start from training with the lowest precision in the precision range of CPT, and then enable CPT at different epochs to see the inﬂuence on the ﬁnal accuracy. Speciﬁcally, we train
ResNet-38/74 on CIFAR-100 for 160 epochs, considering two CPT precision ranges with different starting epochs to enable CPT as shown in Tab. 7.
We consistently observe that (1) an early starting epoch of CPT leads to a better accuracy on all the considered precision ranges and networks, and (2) even enabling CPT at a later training stage still leads to a notable better accuracy than training without CPT.
C A BLATION STUDY : CPT ON TOP OF GRADIENTS
[Table] We decide not to apply CPT on top of gradients since (1) the resulting instability of low precision gradients during training (Wang et al., 2018a) can harm the ﬁnal convergence, and (2) generally the required precision of gradients is higher than that of weights and activations, so that the beneﬁt of applying
CPT on top of gradients in terms of efﬁciency is limited.
To validate this, we show the results of applying CPT on top of gradients with ﬁxed precision for weights and activations in Tab. 8.
As expected, CPT on top of gradients can hardly beneﬁt either accuracy or efﬁciency as compared with its static precision baseline.
Therefore, we decide to adopt ﬁxed precision for gradients in all other experiments.