Multi-Precision Policy Enforced Training (MuPPET) : A precision-switching strategy for quantised fixed-point training of CNNs
Authors: Aditya Rajagopal, Diederik A. Vink, Stylianos I. Venieris, Christos-Savvas Bouganis
Abstract: Large-scale convolutional neural networks (CNNs) suffer from very long training times,  spanning from hours to weeks, limiting the productivity and experimentation of deep learning practitioners.  As networks grow in size and complexity one approach  of  reducing  training  time  is  the  use  of  low-precision  data  representation and computations during the training stage.  However, in doing so the final accuracy suffers due to the problem of vanishing gradients.  Existing state-of-the-art methods combat this issue by means of a mixed-precision approach employing two different precision levels, FP32 (32-bit floating-point precision) and FP16/FP8 (16-/8-bit floating-point precision), leveraging the hardware support of recent GPU architectures for FP16 operations to obtaining performance gains. This work pushes the boundary of quantised training by employing a multilevel optimisation approach that utilises multiple precisions including low-precision fixed-point representations.  The training strategy, named MuPPET, combines the use of  multiple  number  representation regimes  together  with  a  precision-switching mechanism that decides at run time the transition between different precisions. Overall, the proposed strategy tailors the training process to the hardware-level capabilities of the utilised hardware architecture and yields improvements in training time and energy efficiency compared to state-of-the-art approaches. Applying MuPPET on the training of AlexNet, ResNet18 and GoogLeNet on ImageNet (ILSVRC12) and targeting an NVIDIA Turing GPU, the proposed method achieves the same accuracy as the standard full-precision training with an average training-time speedup of 1.28× across the networks.