Paper:
Published as a conference paper at ICLR 2018 MIXED PRECISION TRAINING
Increasing the size of a neural network typically improves accuracy but also increases the memory and compute requirements for training the model. We introduce methodology for training deep neural networks using half-precision floating point numbers, without losing model accuracy or having to modify hyperparameters. This nearly halves memory requirements and, on recent GPUs, speeds up arithmetic. Weights, activations, and gradients are stored in IEEE halfprecision format. Since this format has a narrower range than single-precision we propose three techniques for preventing the loss of critical information. Firstly, we recommend maintaining a single-precision copy of weights that accumulates the gradients after each optimizer step (this copy is rounded to half-precision for the forward-and back-propagation). Secondly, we propose loss-scaling to preserve gradient values with small magnitudes. Thirdly, we use half-precision arithmetic that accumulates into single-precision outputs, which are converted to halfprecision before storing to memory. We demonstrate that the proposed methodology works across a wide variety of tasks and modern large scale (exceeding 100 million parameters) model architectures, trained on large datasets.

Review:
The paper considers the problem of training neural networks in mixed precision (MP), using both 16-bit floating point (FP16) and 32-bit floating point (FP32). The paper proposes three techniques for training networks in mixed precision: first, keep a master copy of network parameters in FP32; second, use loss scaling to ensure that gradients are representable using the limited range of FP16; third, compute dot products and reductions with FP32 accumulation. 
Using these techniques allows the authors to match the results of traditional FP32 training on a wide variety of tasks without modifying any training hyperparameters. The authors show results on ImageNet classification (with AlexNet, VGG, GoogLeNet, Inception-v1, Inception-v3, and ResNet-50), VOC object detection (with Faster R-CNN and Multibox SSD), speech recognition in English and Mandarin (with CNN+GRU), English to French machine translation (with multilayer LSTMs), language modeling on the 1 Billion Words dataset (with a bigLSTM), and generative adversarial networks on CelebFaces (with DCGAN).
Pros:
- Three simple techniques to use for mixed-precision training
- Matches performance of traditional FP32 training without modifying any hyperparameters
- Very extensive experiments on a wide variety of tasks
Cons:
- Experiments do not validate the necessity of FP32 accumulation
- No comparison of training time speedup from mixed precision
With new hardware (such as NVIDIA’s Volta architecture) providing large computational speedups for MP computation, I expect that MP training will become standard practice in deep learning in the near future. Naively porting FP32 training recipes can fail due to the reduced numeric range of FP16 arithmetic; however by adopting the techniques of this paper, practitioners will be able to migrate their existing FP32 training pipelines to MP without modifying any hyperparameters. I expect these techniques to be hugely impactful as more people begin migrating to new MP hardware.
The experiments in this paper are very exhaustive, covering nearly every major application of deep learning. Matching state-of-the-art results on so many tasks increases my confidence that I will be able to apply these techniques to my own tasks and architectures to achieve stable MP training.
My first concern with the paper is that there are no experiments to demonstrate the necessity of FP32 accumulation. With an FP32 master copy of the weights and loss scaling, can all arithmetic be performed solely in FP16, or are there some tasks where training will still diverge?
My second concern is that there is no comparison of training-time speedup using MP. The main reason that MP is interesting is because new hardware promises to accelerate it. If people are willing to endure the extra engineering overhead of implementing the techniques from this paper, what kind of practical speedups can they expect to see from their workloads? NVIDIA’s marketing material claims that the Tensor Cores in the V100 offer an 8x speedup over its general-purpose CUDA cores (https://www.nvidia.com/en-us/data-center/tesla-v100/). Since in this paper some operations are performed in FP32 (weight updates, batch normalization) and other operations are bound by memory and not compute bandwidth, what kinds of speedups do you see in practice when moving from FP32 to MP on V100?
My other concerns are minor. Mandarin speech recognition results are reported on “our internal test set”. Is there any previously published work on this dataset, or any publicly available test set for this task?
The notation around the Inception architectures should be clarified. According to [3] and [4], “Inception-v1” and “GoogLeNet” both refer to the architecture used in [1]. The architecture used in [2] is referred to as “BN-Inception” by [3] and “Inception-v2” by [4]. “Inception-v3” is the architecture from [3], which is not currently cited. To improve clarity in Table 1, I suggest renaming “GoogLeNet” to “Inception-v1”, changing “Inception-v1” to “Inception-v2”, and adding explicit citations to all rows of the table.
In Section 4.3 the authors note that “half-precision storage format may act as a regularizer during training”. Though the effect is most obvious from the speech recognition experiments in Section 4.3, MP also achieves slightly higher performance than baseline for all ImageNet models but Inception-v1 and for both object detection models; these results add support to the idea of FP16 as a regularizer.
Minor typos:
Section 3.3, Paragraph 3: “either FP16 or FP16 math” -> “either FP16 or FP32 math”
Section 4.1, Paragraph 4: “ pre-ativation” -> “pre-activation”
Overall this is a strong paper, and I believe that it will be impactful as MP hardware becomes more widely used.
References
[1] Szegedy et al, “Going Deeper with Convolutions”, CVPR 2015
[2] Ioffe and Szegedy, “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift”, ICML 2015
[3] Szegedy et al, “Rethinking the Inception Architecture for Computer Vision”, CVPR 2016
[4] Szegedy et al, “Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning”, ICLR 2016 Workshop