Paper:
Large-scale convolutional neural networks (CNNs) suffer from very long training times, spanning from hours to weeks, limiting the productivity and experimentation of deep learning practitioners. As networks grow in size and complexity, training time is reduced through low-precision data representations and computations. However, in doing so the final accuracy suffers due to the problem of vanishing gradients. Existing state-of-the-art methods combat this issue by means of a mixed-precision approach utilising two different precision levels, FP32 (32-bit floating-point precision) and FP16/FP8 (16-/8-bit floating-point precision), leveraging the hardware support of recent GPU architectures for FP16 operations to obtain performance gains.This work pushes the boundary of quantised training by employing a multilevel optimisation approach that utilises multiple precisions including low-precision fixed-point representations. The novel training strategy, MuPPET, combines the use of multiple number representation regimes together with a precision-switching mechanism that decides at run time the transition point between precision regimes. Overall, the proposed strategy tailors the training process to the hardware-level capabilities of the utilised hardware architecture and yields improvements in training time and energy efficiency compared to state-ofthe-art approaches. Applying MuPPET on the training of AlexNet, ResNet18 and GoogLeNet on ImageNet (ILSVRC12) and targeting an NVIDIA Turing GPU, MuPPET achieves the same accuracy as standard full-precision training with an average training-time speedup of 1.28× across the networks.

Review:
Summary:
This paper proposes a training strategy called Multi-Precision Policy Enforced Training(MUPPET). This strategy aims to reduce training time by low-precision data representation and computations during the training stage. According to the gradient diversity, the authors introduce a precision-switching mechanism which chooses the best epoch to increase the precision. The validation accuracy and training time across several networks and datasets are shown in the experiments. However, the results are not superior enough compared with the state-of-the-art.
My detailed comments are as follows.
Positive points: 
1. This paper proposes a new reduced-precision training scheme to speed up training by progressively increasing the precision of computations from 8-bit fixed-point to 32-bit floating-point. This scheme moves to reduced-precision fixed-point computations while updating an FP32 model in order to push the boundaries of reduced-precision training. 
2. The authors propose a metric to decide when to switch the precision inspired by gradient diversity introduced by [1]. In this paper, the gradient diversity is enhanced by considering gradients across epochs instead of mini-batches. The proposed metric can be seen as a proxy for the amount of new information gained in each training step. Therefore, the metric can decide the most appropriate epoch at run time to increase the precision.
 
3. The proposed low-precision CNN training scheme is orthogonal and complementary to existing low-precision training techniques.

Negative points:
1. The proposed approach does not match the description in this paper. The authors describe “This approach enables the design of a policy that can decide at run time the most appropriate quantization level for the training process”. In fact, this approach just decides which epoch to increase the quantization level while the levels of quantized precisions are fixed, rather than deciding the most appropriate quantization level. 
2. The setting of quantized precision levels (8-, 12-, 14- and 16-bit precisions) is confusing. Please illustrate how to choose the number of quantized bit and the number of quantized precision levels.
3. The presentation of the precision switching policy is confusing and the notations are unclear. For example, in section 3.3, the ratio “p” needs more description because it is a key value in the policy, but lacks an explanation in this section. So please explain more about the motivation of ratio “p” in this section. 	In section 3.3, in step 5 of the proposed precision switching policy, the authors do not explain the meaning of “y”.
4. In figure 2, the precision switch is not triggered even though the value of p violates the threshold more than 2 times, which mismatches the description in section 3.3.
5. The proposed strategy has no obvious advantages. There are some scenes that the proposed strategy does not perform well. For example, the Top-1 validation accuracy on ImageNet of AlexNet and ResNet with MuPPET strategy is much lower than FP32 baseline. Compared with [2], the proposed method is more complex but not superior enough.
6. The authors do not show the training and validation curves. However, the training and validation curves are common used to show more details of the training process, such as in [2] and [3]. Please show and analyze the training and validation curves of the proposed scheme and the baseline.
Minor issues:
Some spelling and grammar mistakes.
Reference：
[1] Dong Yin, Ashwin Pananjady, Max Lam, Dimitris Papailiopoulos, Kannan Ramchandran, and Peter Bartlett. Gradient Diversity: a Key Ingredient for Scalable Distributed Learning. In 21st International Conference on Artificial Intelligence and StatiZZstics (AISTATS), pp. 1998–2007, 2018.
[2] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed Precision Training. In International Conference on Learning Representations (ICLR), 2018.
[3]  Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep Learning with Limited Numerical Precision. In 32nd International Conference on Machine Learning (ICML), pp. 1737–1746, 2015.