Original: M ULTI -P RECISION
Revised: M ULTI -P RECISION

Original: P OLICY
Revised: P OLICY

Original: E NFORCED
Revised: E NFORCED

Original: T RAINING
Revised: T RAINING

Original: (M U PPET) : A PRECISION - SWITCHING STRATEGY FOR QUANTISED FIXED - POINT TRAINING OF CNN S
Revised: (M U PPET) : A PRECISION - SWITCHING STRATEGY FOR QUANTISED FIXED - POINT TRAINING OF CNN S

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Large-scale convolutional neural networks (CNNs) suffer from very long training times, spanning from hours to weeks, limiting the productivity and experimentation of deep learning practitioners.
Revised: Large-scale convolutional neural networks (CNNs) suffer from very long training times, spanning from hours to weeks, limiting the productivity and experimentation of deep learning practitioners.

Original: As networks grow in size and complexity,one approach of reducing training time is the use of low-precision data representation and computations during the training stage.
Revised: As networks grow in size and complexity, training time is reduced through low-precision data representations and computations.

Original: However, in doing so the ﬁnal accuracy suffers due to the problem of vanishing gradients.
Revised: However, in doing so the ﬁnal accuracy suffers due to the problem of vanishing gradients.

Original: Existing stateof-the-art methods combat this issue by means of a mixed-precision approach employing two different precision levels, FP32 (32-bit ﬂoating-point precision) and FP16/FP8 (16-/8-bit ﬂoating-point precision), leveraging the hardware support of recent GPU architectures for FP16 operations to obtaining performance gains.
Revised: Existing state-of-the-art methods combat this issue by means of a mixed-precision approach utilising two different precision levels, FP32 (32-bit ﬂoating-point precision) and FP16/FP8 (16-/8-bit ﬂoating-point precision), leveraging the hardware support of recent GPU architectures for FP16 operations to obtain performance gains.

Original: This work pushes the boundary of quantised training by employing a multilevel optimisation approach that utilises multiple precisions including low-precision ﬁxedpoint representations.
Revised: This work pushes the boundary of quantised training by employing a multilevel optimisation approach that utilises multiple precisions including low-precision ﬁxed-point representations.

Original: The  training strategy,named MuPPET, combines the use of multiple number representation regimes together with a precision-switching mechanism that decides at run time the transition  between different precisions.
Revised: The novel training strategy, MuPPET, combines the use of multiple number representation regimes together with a precision-switching mechanism that decides at run time the transition point between precision regimes.

Original: Overall, the proposed strategy tailors the training process to the hardware-level capabilities of the utilised hardware architecture and yields improvements in training time and energy efﬁciency compared to state-of-the-art approaches.
Revised: Overall, the proposed strategy tailors the training process to the hardware-level capabilities of the utilised hardware architecture and yields improvements in training time and energy efﬁciency compared to state-ofthe-art approaches.

Original: Applying MuPPET on the training of AlexNet, ResNet18 and GoogLeNet on ImageNet (ILSVRC12) and targeting an NVIDIA Turing GPU, the proposed method achieves the same accuracy asthe standard full-precision training with an average training-time speedup of 1.28 × across the networks.
Revised: Applying MuPPET on the training of AlexNet, ResNet18 and GoogLeNet on ImageNet (ILSVRC12) and targeting an NVIDIA Turing GPU, MuPPET achieves the same accuracy as standard full-precision training with an average training-time speedup of 1.28 × across the networks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Convolutional neural netwoks (CNNs) have demonstrated unprecedented accuracy in various machine learning tasks, from video understanding (Karpathy et al., 2014; Gan et al., 2015;
Revised: Convolutional neural networks (CNNs) have demonstrated unprecedented accuracy in various machine learning tasks, from video understanding (Gan et al., 2015;

Original: He et al., 2018)and action recognition (Simonyan & Zisserman, 2014; Ng et al., 2015; Varol et al., 2018) toautonomous drone navigation (Gandhi et al., 2017; Loquercio et al., 2018; Kouris & Bouganis, 2018).
Revised: He et al., 2018) to drone navigation (Loquercio et al., 2018; Kouris & Bouganis, 2018).

Original: To achieve such high levels of accuracy in inherently complex applications, currentdeep learning methodologies employ the design of large and complex CNN models (He et al., 2016; Szegedy et al., 2017; Huang et al., 2017) trained over large datasets (Deng et al., 2009;Geiger et al., 2013; Lin et al., 2014).
Revised: To achieve such high levels of accuracy in inherently complex applications, current methodologies employ the design of large and complex CNN models (Szegedy et al., 2017; Huang et al., 2017) trained over large datasets (Deng et al., 2009; Lin et al., 2014).

Original: Nevertheless, the combination of large models and massivetraining datasets leads to long training times.
Revised: Nevertheless, the combination of large models and massive datasets results in long training times.

Original: This in turn leads to long turn-around times which limits the productivity of deep learning practitioners and prohibits wider experimentation with different model conﬁgurations.
Revised: This in turn leads to long turn-around times which limits the productivity of deep learning practitioners and prohibits wider experimentation.

Original: For instance, automatic tuning and search of neural architectures (Cai et al., 2018; Zhong et al., 2018) is a rapidly advancing area where accelerated training enables improving the produced networks.
Revised: For instance, automatic tuning and search of neural architectures (Cai et al., 2018; Zhong et al., 2018) is a rapidly advancing area where accelerated training enables improving the produced networks.

Original: To counteract these long turn-around times, substantial research effort has been invested in hyperparameter tuning for training acceleration, with a particular focus on batch size and content.
Revised: To counteract these long turn-around times, substantial research effort has been invested in hyperparameter tuning for the acceleration of training, with a particular focus on batch size and content.

Original: One line of work maximises memory and hardware utilisation by changing the batch size, in order to perform CNN-training-speciﬁc prefetching, scheduling and dependency improvement (Chen et al., 2019; Rhu et al., 2016).
Revised: One line of work maximises memory and hardware utilisation by changing the batch size, in order to perform CNN training speciﬁc prefetching, scheduling and dependency improvement (Chen et al., 2019; Rhu et al., 2016).

Original: Other works focus on altering batch size  to improve the convergence rate while maintaining high hardware utilisation (Devarakonda et al., 2017) and demonstrating up to 6.25 × training acceleration.
Revised: Other works focus on altering batch size or reconstructing minibatches to improve the convergence rate while sustaining high hardware utilisation (Devarakonda et al., 2017;

Original: Alternatively, works such as (Johnson & Guestrin, 2018;Katharopoulos & Fleuret, 2018; Peng et al., 2019; Loshchilov & Hutter, 2016) reconstruct the minibatches according to the predicted impact of each data element to improve the rate of training convergence, resulting in up to 1.2 × overall acceleration.
Revised: Johnson & Guestrin, 2018; Peng et al., 2019), demonstrating up to 6.25 × training speedup.

Original: A number of studies have focused on the use of reduced-precision training schemes.
Revised: A number of studies have focused on the use of reduced-precision training schemes.

Original: Reducedprecision arithmetic involves the utilisation of data formats that have lower wordlengths than the conventional 32-bitsingle-precision ﬂoating-point (FP32) representation and is an approach for cooptimising processing speed, memory footprint and communication overhead.
Revised: Reducedprecision arithmetic involves the utilisation of data formats that have smaller wordlengths than the conventional 32-bit ﬂoating-point (FP32) representation and is an approach for co-optimising pro- cessing speed, memory footprint and communication overhead.

Original: Existing literature can be categorised into works that use reduced precision to accelerate only the training stage while targeting an FP32 model, and those that target networks with quantised weights.
Revised: Existing literature can be categorised into works that use reduced precision to accelerate only the training stage while targeting an FP32 model, and those that produce networks with quantised weights.

Original: Amongst those that target FP32 networks, one line of work involves utilising different quantisation strategies such as dynamic quantisation(Courbariaux et al., 2015) and stochastic rounding (Gupta et al., 2015) as a means to combat the accuracy loss from quantisation.
Revised: Regarding the former, Courbariaux et al. (2015) and Gupta et al. (2015) utilise dynamic quantisation and stochastic rounding respectively as a means to combat the accuracy loss due to quantisation.

Original: Nevertheless, the effectiveness of the proposed schemes has been demonstrated on small-scale datasets such as CIFAR-10 and MNIST, and on a limited set of networks.
Revised: Nevertheless, the effectiveness of the proposed schemes have only been demonstrated on smallscale datasets such as CIFAR-10 and MNIST, and on a limited set of networks.

Original: Furthermore, the range of quantisation levels that has been explored varies greatly, with a number of works attacking the problem from an algorithmic level and focusing on mild quantisation levels such as half-precision ﬂoating-point (FP16) (Micikevicius et al., 2018), while others focus on lower precisions such as 8-bit ﬂoating-point (FP8) (Wang et al., 2018).
Revised: Furthermore, the range of quantisation levels that has been explored varies greatly, with a number of works attacking the problem by focusing on mild quantisation levels such as half-precision ﬂoating-point (FP16) (Micikevicius et al., 2018), while others focus on lower precisions such as 8-bit ﬂoating-point (FP8) (Wang et al., 2018).

Original: Finally, quantisation has also been used as a means of reducing the memory and communication overhead in distributed trainingsetups (De Sa et al., 2015; 2017; Alistarh et al., 2017).
Revised: Finally, quantisation has also been used as a means of reducing the memory and communication overhead in distributed training (De Sa et al., 2015; 2017; Alistarh et al., 2017).

Original: At the same time, the characteristics of modern CNN workloads and the trend towards precisionquantised models have led to an emergence of specialised hardware processors, featuring support for low-precision arithmethic at the hardware level.
Revised: At the same time, the characteristics of modern CNN workloads and the trend towards quantised models have led to an emergence of specialised hardware processors, with support for low-precision arithmetic at the hardware level.

Original: Spanning from custom designs such  Google’s TPUs (Jouppi et al., 2017) and Microsoft’s FPGA-based Brainwave system (Fowers et al., 2018) to commodity platforms such as NVIDIA’s Turing GPUs, existingprocessing platforms include hardware support for reduced precision including 16-bithalf-precision ﬂoating-point (FP16), and 8-bit (INT8) and 4-bit (INT4) ﬁxed-point data types, thus providing increased parallelism for lower bitwidths.
Revised: From custom designs such as Google’s TPUs (Jouppi et al., 2017) and Microsoft’s FPGA-based Brainwave system (Fowers et al., 2018) to commodity devices such as NVIDIA’s Turing GPUs, existing platforms offer native support for reduced-precision data types including 16-bit ﬂoating-point (FP16), and 8- (INT8) and 4-bit (INT4) ﬁxed-point, providing increased parallelism for lower bitwidths.

Original: Although these platforms have been mainly designed to target the inference stage of already-trained networks, the reduced-precision hardware offers signiﬁcant opportunities for accelerating the timeconsuming training stage.
Revised: Although these platforms have been mainly designed for the inference stage, the low-precision hardware offers signiﬁcant opportunities for accelerating the time-consuming training stage.

Original: In this respect, there is an emerging need to provide training algorithms that can leverage these existing hardware optimisations and provide higher training speed.
Revised: In this respect, there is an emerging need to provide training algorithms that can leverage these existing hardware optimisations and provide higher training speed.

Original: This work focuses on tackling the ﬁeld of reduced-precision training from an algorithmic level.
Revised: This work tackles the ﬁeld of reduced-precision training at an algorithmic level.

Original: Independently of the number of quantisation levels chosen, or how extreme the quantisation is, this work proposes a metric that estimates the amount of information each new training step obtains for a given quantisation level.
Revised: Independently of the number of quantisation levels chosen, or how extreme the quantisation is, this work proposes a metric that estimates the amount of information each new training step obtains for a given quantisation level, by capturing the diversity of the computed gradients across epochs .

Original: Thisapproach enables the design of a policy that can decide at run timethe most appropriate quantisation level for the training process and, due to its agnostic nature, it is orthogonal and complementary to existing low-precision training techniques.
Revised: This enables the design of a policy that, given a set of quantisation levels, decides at run time appropriate points to increase the precision of the training process at that current instant without impacting the achieved test accuracy compared to training in FP32. Due to its agnostic nature, it remains orthogonal and complementary to existing low-precision training schemes.

Revised: Furthermore, by pushing the precision below the 16-bit bitwidth of existing state-of-the-art techniques, the proposed method is able to leverage the lowprecision capabilities of modern processing systems to yield training speedups without penalising the resulting accuracy, signiﬁcantly improving the time-to-accuracy trade-off.

Original:  B ACKGROUND AND R ELATED W ORK
Revised:  B ACKGROUND AND R ELATED W ORK

Original: The state-of-the-art method in training in reduced precision is mixed-precision training (Micikevicius et al., 2018).
Revised: The state-of-the-art method in training in reduced precision is mixed-precision training (Micikevicius et al., 2018).

Original: The authors propose to employ low-precision FP16 computations in the training stage of high-precision CNNs thatwould perform inference in FP32.
Revised: The authors propose to employ low-precision FP16 computations in the training stage of high-precision CNNs that perform inference in FP32.

Original: Along the training phase, the algorithm maintains a high-precision FP32 copy of the weights of the network, known as a master copy .
Revised: Along the training phase, the algorithm maintains a high-precision FP32 copy of the weights of the network, known as a master copy .

Original: At each minibatch, the inputs and weights are quantised to FP16 with all computations of the forward and backward pass performed in FP16, yielding memory footprint and runtime savings.
Revised: At each minibatch, the inputs and weights are quantised to FP16 with all computations of the forward and backward pass performed in FP16, yielding memory footprint and runtime savings.

Original: Under this scheme, each stochastic gradient descent (SGD) update step entails accumulating FP16 gradients into the FP32 master copy of the weights, with this process performed iteratively throughout the training of the network.
Revised: Under this scheme, each stochastic gradient descent (SGD) update step entails accumulating FP16 gradients into the FP32 master copy of the weights, with this process performed iteratively throughout the training of the network.

Original: Micikevicius et al.
Revised: Micikevicius et al.

Original: (2018) evaluate their scheme over a set of state-of-the-art models on ImageNet, and show that mixed-precision training with FP16 computations achieves comparable accuracy to standard FP32 training.
Revised: (2018) evaluate their scheme over a set of state-of-theart models on ImageNet, and show that mixed-precision training with FP16 computations achieves comparable accuracy to standard FP32 training.

Original: Wang et al.
Revised: Wang et al.

Original: (2018) also presented a method to train an FP32 model using 8-bit ﬂoating-point (FP8).
Revised: (2018) also presented a method to train an FP32 model using 8-bit ﬂoating-point (FP8).

Original: The authors propose a hand-crafted FP8 data type, together with a chunk-based computation technique, and employ strategies such as stochastic rounding to alleviate the accuracy loss due to training at reduced precision.
Revised: The authors propose a hand-crafted FP8 data type, together with a chunk-based computation technique, and employ strategies such as stochastic rounding to alleviate the accuracy loss due to training at reduced precision.

Original: For AlexNet, ResNet18 and ResNet50 on ImageNet, Wang et al.
Revised: For AlexNet, ResNet18 and ResNet50 on ImageNet, Wang et al.

Original: (2018) demonstrates comparable accuracy to FP32 training while performing computations in FP8.
Revised: (2018) demonstrates comparable accuracy to FP32 training while performing computations in FP8.

Original: Additionally the works presented in (Zhou et al., 2016; Chen et al., 2017) approach the problem of reduced-precision training employing ﬁxed-point computations.
Revised: Additionally the works presented in (Zhou et al., 2016; Chen et al., 2017) approach the problem of reduced-precision training employing ﬁxed-point computations.

Original: FxpNet (Chen et al., 2017) was only evaludated on CIFAR-10, failing to demonstrate performance on more complex datasets such as ImageNet.
Revised: FxpNet (Chen et al., 2017) was only evaluated on CIFAR-10, failing to demonstrate performance on more complex datasets such as ImageNet.

Original: DoReFa-net (Zhou et al., 2016)  tested on ImageNet but only ran on AlexNet missing out on state-of-the-art networks such as GoogLeNet and ResNet.
Revised: DoReFa-net (Zhou et al., 2016) was tested on ImageNet but only ran on AlexNet missing out on state-of-the-art networks such as GoogLeNet and ResNet.

Original: All related works focus on accelerating  training  an FP32 model through reduced-precision computations.
Revised: All related works focus on accelerating the training of an FP32 model through reduced-precision computations.

Original: At the hardware level, 8-bit ﬁxed-point multiplication uses 18.5 × less energy and 27.5 × less area with up to 4 × lower multiplication times than FP32 (Sze et al., 2017).
Revised: At the hardware level, 8-bit ﬁxed-point multiplication uses 18.5 × less energy and 27.5 × less area with up to 4 × lower multiplication times than FP32 (Sze et al., 2017).

Original: Consequently, this work attempts to push the boundaries of reduced-precision training by moving to reduced-precision ﬁxed-point computations while updating an FP32 model.
Revised: Consequently, this work attempts to push the boundaries of reduced-precision training by moving to reducedprecision ﬁxed-point computations while updating an FP32 model.

Original: Preliminary tests on training AlexNet and ResNet20 on CIFAR-100 using 8-bit ﬁxed-point computations to update an FP32 model demonstrated that training solely in 8-bit ﬁxed-point results in a signiﬁcant degradation of validation accuracy compared to full FP32 training (Table 1).
Revised: Preliminary tests (Sec. 4.4 for details) demonstrated that training solely in 8-bit ﬁxed-point results in a signiﬁcant degradation of validation accuracy compared to full FP32 training.

Original: This work aims to counteract this degradation by progressively increasing the precision of computations throughout training in an online manner determined by the proposed metric inspired by gradient diveristy (Yin et al., 2018).
Revised: This work aims to counteract this degradation by progressively increasing the precision of computations throughout training in an online manner determined by the proposed metric inspired by gradient diversity (Yin et al., 2018).

Original: Additionally by operating in an online fashion, MuPPET tailors the training process to best suit the particular network-dataset pair at each stage of the training process.
Revised: Additionally by operating in an online fashion, MuPPET tailors the training process to best suit the particular network-dataset pair at each stage of the training process.

Original: Gradient diversity was introduced by Yin et al.
Revised: Gradient diversity was introduced by Yin et al.

Original: as a metric of measuring the dissimilarity between sets of gradients that correspond to different minibatches.
Revised: as a metric of measuring the dissimilarity between sets of gradients that correspond to different minibatches.

Original: The gradient diversity of a set of gradients is deﬁned as
Revised: The gradient diversity of a set of gradients is deﬁned as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  where ∇ f i ( w ) represents the gradient of weights w for minibatch i .
Revised: ) where ∇ f i ( w ) represents the gradient of weights w for minibatch i .

Original: Yin et al.

Original: (2018) argue that if the gradients obtained from consecutive minibatches are not diverse enough, then the performance beneﬁts of scaling the system to multiple processors do not scale linearly with the number of processors and, hence, propose the use of this metric for choosing the optimal minibatch size for a system.

Original: The key point to note in Eq.
Revised: The key point to note in Eq.

Original: (1) is that the denominator contains the inner product between two gradients from different minibatches .
Revised: (1) is that the denominator contains the inner product between two gradients from different minibatches .

Original: Thus, orthogonal gradients would result in high gradient diversity, while similar gradients would result in low gradient diversity.
Revised: Thus, orthogonal gradients would result in high gradient diversity, while similar gradients would result in low gradient diversity.

Original: The proposed framework, MuPPET, enhances this concept by considering gradients  across epochsinstead of minibatches and proposes the developed metric as a proxy for the amount of new information gained in each training step.
Revised: The proposed framework, MuPPET, enhances this concept by considering gradients between minibatches across epochs and proposes the developed metric as a proxy for the amount of new information gained in each training step.

Original: Section 3 further expands on howthe idea of gradient diversity is incorporated into the MuPPET algorithm.
Revised: Section 3 further expands on how gradient diversity is incorporated into the MuPPET algorithm.

Original:  M ETHODOLOGY 3. M ULTILEVEL OPTIMISATION FOR TRAINING CNN S
Revised:  M ETHODOLOGY 3. M ULTILEVEL OPTIMISATION FOR TRAINING CNN S

Original: Conventionally, the training process of a CNN can be expressed as in Eq.
Revised: Conventionally, the training process of a CNN can be expressed as in Eq.

Original: Given a CNN model f parameterised with respect to a set of weights w ∈ R D , where D is the number of weights of f , training involves a search for weight values that minimise the task-speciﬁc empirical loss, Loss , on the target dataset.
Revised: Given a CNN model f parameterised by a set of weights w ∈ R D , where D is the number of weights of f , training involves a search for weight values that minimise the task-speciﬁc empirical loss, Loss , on the target dataset.

Original: Typically, a ﬁxed arithmetic precision is employed across the training algorithm with FP32 currently being the de facto representation used by the deep learning community.
Revised: Typically, a ﬁxed arithmetic precision is employed across the training algorithm with FP32 currently being the de facto representation used by the deep learning community.

Original: [Equation]
Revised: [Equation]

Original: The proposed method follows a different approach by introducing a multilevel optimisation scheme (Migdalas et al., 2013) that leverages the performance gains of reduced-precision arithmetic.
Revised: The proposed method follows a different approach by introducing a multilevel optimisation scheme (Migdalas et al., 2013) that leverages the performance gains of reduced-precision arithmetic.

Original: The single optimisation problem of Eq.
Revised: The single optimisation problem of Eq.

Original: (2) is transformed into a series of optimisation problems with each one employing different precision for computations, but maintaining weights storage at FP precision.
Revised: (2) is transformed into a series of optimisation problems with each one employing different precision for computations, but maintaining weights storage at FP precision.

Original: Under this scheme, an N -level formulation comprises N sequential optimisation problems to be solved, with each level corresponding to a “ﬁner” model.
Revised: Under this scheme, an N -level formulation comprises N sequential optimisation problems to be solved, with each level corresponding to a “ﬁner” model.

Original: Overall, this formulation adds a hierarchical structure to the training stage, with an increasing arithmetic precision across the hierarchy of optimisation problems.
Revised: Overall, this formulation adds a hierarchical structure to the training stage, with an increasing arithmetic precision across the hierarchy of optimisation problems.

Original: Starting from the N -th problem, the inputs, weights, and activations of the CNN model f are quantised with precision q N , which is the lowest precision in the system and represents the coarsest version of the model.
Revised: Starting from the N -th problem, the inputs, weights, and activations of the CNN model f are quantised with precision q N , which is the lowest precision in the system and represents the coarsest version of the model.

Original: Each of the N levels progressively employs higher precision until the ﬁrst level is reached, which corresponds to the original problem of Eq.
Revised: Each of the N levels progressively employs higher precision until the ﬁrst level is reached, which corresponds to the original problem of Eq.

Original: Formally, at the i-th level, the optimisation problem is formulated as
Revised: Formally, at the i-th level, the optimisation problem is formulated as

Original: [Equation]
Revised: [Equation]

Original: ∈ represent. range( q i [Equation]
Revised: where LB and UB are the lower and upper bound in the representational range of precision q i .

Original: The target CNN model f uses a set of weights quantised with precision q i and hence the solution of this optimisation problem can be interpreted as an approximation to the original problem of Eq.
Revised: The target CNN model f uses a set of weights quantised with precision q i and hence the solution of this optimisation problem can be interpreted as an approximation to the original problem of Eq.

Original: To transition from one level to the next, the result of each level of optimisation is employed as a starting point for the next level, up to the ﬁnal outermost optimisation that reduces to Eq.
Revised: To transition from one level to the next, the result of each level of optimisation is employed as a starting point for the next level, up to the ﬁnal outermost optimisation that reduces to Eq.

Original: 3. T HE M U PPET A LGORITHM
Revised: [Figure]

Original: Fig.
Revised: Fig.

Original: 1 presents the process of training a CNN using the proposed algorithm.
Revised: 1 presents the process of training a CNN using the proposed algorithm.

Revised: All ﬁgures in this paper are shown in the Appendix B at a larger scale for enhanced readability.

Original: Within each epoch, MuPPET performs mixed-precision training where the weights are stored in an FP32 master copy and are quantised to the desired ﬁxed-point precision on-the-ﬂy prior to execution.
Revised: Within each epoch, MuPPET performs mixed-precision training where the weights are stored in an FP32 master copy and are quantised to the desired ﬁxed-point precision on-the-ﬂy.

Original: At epoch j , the computations for the forward and backward passes ( F and B blocks respectively) are performed at the current quantised precision ( q j ) and the activations as well as the gradients obtained from each layer are quantised by the quantiser module before being passed on to the next layer, or stored.
Revised: At epoch j , the computations for the forward and backward passes ( F and B blocks respectively) are performed at the current quantised precision ( q j ) and the activations as well as the gradients obtained from each layer are quantised by the quantiser module before being passed on to the next layer, or stored.

Original: At the end of each epoch, the full-precision master copy of the weights is updated using a quantised gradient matrix.
Revised: After each minibatch, the full-precision master copy of the weights is updated using a quantised gradient matrix.

Original: As discussed in Section 3.1, the quantisation level is gradually increased over the period of the training.
Revised: As discussed in Section 3.1, the quantisation level is gradually increased over the period of the training.

Original: In the proposed multilevel optimisation formulation, switching between these optimisation levels at the correct times is crucial in order not to compromise the ﬁnal validation accuracy.
Revised: In MuPPET, switching between these optimisation levels at the correct times is crucial in order not to compromise the ﬁnal validation accuracy.

Original: In this respect, MuPPET introduces a precision switching policy based on anenhanced inter-epoch gradient diversity (Yin et al., 2018) metric that dictates when to switch to the next precision in order not to degrade the ﬁnal resulting accuracy.
Revised: In this respect, MuPPET introduces a precision switching policy based on an inter-epoch gradient diversity (Yin et al., 2018) metric that dictates when to switch to the next precision.

Original: Details of the switching policy are presented in Section 3.3.
Revised: Details of the switching policy are presented in Section 3.3.

Original: 3.2. Q UANTISATION S TRATEGY
Revised: 3.2. Q UANTISATION S TRATEGY

Original: In order to implement quantised training, a quantisation strategy needs to be deﬁned.
Revised: In order to implement quantised training, a quantisation strategy needs to be deﬁned.

Original: The proposed dynamic quantisation strategy utilises block ﬂoating-point arithmetic (also known as dynamic ﬁxedpoint), where each ﬁxed-point number is represented as a pair of an n -bit signed integer  and a scale factor s , such that the value is represented as x × 2 − s .
Revised: The proposed dynamic quantisation strategy utilises block ﬂoating-point arithmetic (also known as dynamic ﬁxedpoint), where each ﬁxed-point number is represented as a pair of an WL net -bit signed integer x and a scale factor s , such that the value is represented as x × 2 − s .

Original: During the forward and backward passes of the training process, the weights and feature maps are both quantised, and the multiplication operations are performed at the same low precision.
Revised: During the forward and backward passes of the training process, the weights and feature maps are both quantised, and the multiplication operations are performed at the same low precision.

Original: The quantisation method employs a stochastic rounding methodology (Gupta et al., 2015).
Revised: The quantisation method employs a stochastic rounding methodology (Gupta et al., 2015).

Original: The accumulation stage of the matrix-multiply operation is accumulated into a 32-bit ﬁxed-point value to prevent overﬂow on the targeted networks.
Revised: The accumulation stage of the matrix-multiply operation is accumulated into a 32-bit ﬁxed-point value to prevent overﬂow on the targeted networks.

Original: The result of this matrix multiplication is subsequently quantised to the desired wordlength before being passed as input to the next layer.
Revised: The result of this matrix multiplication is subsequently quantised to the target wordlength before being passed as input to the next layer.

Original: Following the block ﬂoating-point scheme, quantisation is performed such that each weight and feature map matrix in the network has a single scale factor shared by all values within the matrix.
Revised: Following the block ﬂoating-point scheme, quantisation is performed such that each weight and feature map matrix in the network has a single scale factor shared by all values within the matrix.

Original: The quantisation conﬁguration for the i-th level of optimisation and the  full set of conﬁgurations are given by  Eq.
Revised: The quantisation conﬁguration for the i-th level of optimisation and the l-th layer, q il , and the full set of conﬁgurations, q i , are given by left- and right-hand side of Eq.

Original: (4)and (5) respectively.
Revised: (4) respectively.

Original: [Equation]
Revised: [Equation]

Original: where |L| is the number of layers of the target network, WL net is the ﬁxed wordlength across the network, SC weights l and SC act l are the scaling factors for the weights and activations, respectively, of the l-th layer for the i-th level of optimisation.
Revised: where |L| is the number of layers of the target network, WL net is the ﬁxed wordlength across the network, s weights l and s act l are the scaling factors for the weights and activations respectively, of the lth layer for the i-th level of optimisation.

Original: As a result, for N levels, there are N distinct quantisation schemes; N − 1 of these schemes are with varying ﬁxed-point precisions, and the ﬁnest level of quantisation, q 1 , is single-precision ﬂoating-point (FP32). 
Revised: As a result, for N levels, there are N distinct quantisation schemes; N − 1 of these schemes are with varying ﬁxed-point precisions, and the ﬁnest level of quantisation, q 1 , is single-precision ﬂoating-point (FP32). The scaling factor for a matrix X is ﬁrst calculated as shown in Eq. (5) and individual elements are quantised as in Eq. [Equation]

Original: 3.2. I NFORMATION T RANSFER B ETWEEN L EVELS
Revised: where X { weights, act } { max, min } is either the maximum or minimum value in the weights or feature maps matrix of the current layer, LB and UB are the lower and upper bound of the current wordlength WL net , and Unif(a,b) represents sampling from the uniform distribution in the range [a,b]. Eq. (5) adds 0 .

Revised: 5 and − 0 .

Revised: 5 to UB and LB respectively to ensure maximum utilisation of WL net .

Revised: 3.2. I NFORMATION T RANSFER B ETWEEN L EVELS

Original: Employing multilevel training for CNNs requires an appropriate mechanism for transferring information between levels.
Revised: Employing multilevel training for CNNs requires an appropriate mechanism for transferring information between levels.

Original: To achieve this, the proposed optimiser maintains a master copy of the weights in full precision (FP23) throughout the optimisation levels.
Revised: To achieve this, the proposed optimiser maintains a master copy of the weights in full precision (FP32) throughout the optimisation levels.

Original: Similar to mixed-precision training (Micikevicius et al., 2018), at each level the SGD update step is performed by accumulating a ﬁxed-point gradient value into the FP32 master copy of the weights.
Revised: Similar to mixed-precision training (Micikevicius et al., 2018), at each level the SGD update step is performed by accumulating a ﬁxed-point gradient value into the FP32 master copy of the weights.

Original: Starting from the coarsest quantisation level i = N , to transfer the solution from level i to level i − 1 , the master copy is quantised using the quantisation scheme q i − 1 .
Revised: Starting from the coarsest quantisation level i = N , to transfer the solution from level i to level i − 1 , the master copy is quantised using the quantisation scheme q i − 1 .

Original: With this approach, the weights are maintained in
Revised: With this approach, the weights are maintained in

Original: FP32 and are quantised on-the-ﬂy during run time in order to be utilised in each training step, as w [Equation] (cid:16) [Equation]
Revised: FP32 and are quantised on-the-ﬂy during run time in order to be utilised in each training step.

Original: ( FP32 )

Original: [Equation]

Original: [Equation]

Original: (cid:17) .

Original: 3. P RECISION S WITCHING P OLICY
Revised: 3. P RECISION S WITCHING P OLICY

Original: The metric to decide when to switch between levels of quantisation is inspired by Yin et al.
Revised: The metric to decide when to switch between levels of quantisation is inspired by Yin et al.

Original: and based on the concept of gradient diversity (Eq.
Revised: and based on the concept of gradient diversity (Eq.

Original: As discussed in Section 2, Yin et al.

Original: (2018) compute the gradient diversity between gradients across minibatches. Instead, MuPPET computes ∆ S ( w ) between gradients  across epochsinstead of minibatches as a proxy to measure the information that is obtained during the training process; the lower the diversity between the gradients, the less information this level of quantisation is providing towards the training of the model.
Revised: MuPPET computes ∆ S ( w ) between gradients obtained across epochs as a proxy to measure the information that is obtained during the training process; the lower the diversity between the gradients, the less information this level of quantisation provides towards the training of the model.

Original: Therefore, the proposed method comprises a novel normalised inter-epoch version of the gradient diversity along with a run-time policy to determine the epochs to switch precision.
Revised: Therefore, the proposed method comprises a novel normalised inter-epoch version of the gradient diversity along with a run-time policy to determine the epochs to switch precision.

Original: The following policy is employed to determine when a precision switch is to be performed.
Revised: The following policy is employed to determine when a precision switch is to be performed.

Original: For a network with layers L and a quantisation scheme q i that was switched into at epoch e :
Revised: For a network with layers L and a quantisation scheme q i that was switched into at epoch e :

Original: For each epoch j and each layer l ∈ L , the last  gradient, ∇ f jl ( w ) , is stored.
Revised: For each epoch j and each layer l ∈ L , the last minibatch’s gradient, ∇ f jl ( w ) , is stored.

Original: After r (resolution) number of epochs, the inter-epoch gradient diversity at epoch j is
Revised: After r (resolution) number of epochs, the inter-epoch gradient diversity at epoch j is

Original: [Equation]
Revised: [Equation]

Original: At an epoch j , given a set of gradient diversities S ( j ) =
Revised: At an epoch j , given a set of gradient diversities S ( j ) =

Original: (cid:110) ∆ S ( w ) i
Revised: (cid:110) ∆ S ( w ) i

Original: ∀ e ≤ i < j (cid:111) , the ratio p = max S ( j ) ∆ S ( w ) j is calculated.
Revised: ∀ e ≤ i < j (cid:111) , the ratio p = max S ( j ) ∆ S ( w ) j is calculated.

Original: An empirically determined decaying threshold T =
Revised: An empirically determined decaying threshold T =

Original: α + βe − λj (7) is placed on the ratio p .
Revised: α + βe − λj (8) is placed on the ratio p .

Original: If the p violates y more than γ times, a precision switch is triggered and S ( j ) = ∅ .
Revised: If the p violates T more than γ times, a precision switch is triggered and S ( j ) = ∅ .

Revised: As long as the gradients across epochs remain diverse, ∆ S ( w ) j

Revised: (Eq.(7))

Original: 3.3. H YPERPARAMETERS
Revised: at the denominator of p sustains a high value and the value of p remains low. However, when the gradients across epochs become similar, ∆ S ( w ) j decreases and the value of p becomes larger. Generalisability across epochs is obtained as p accounts for the change in information relative to the maximum information available since the last precision change. Hence, the metric acknowledges the presence of temporal variations in information provided by the gradients.

Revised: Generalisability across networks and datasets is maintained as p measures a ratio.

Revised: Consequently, the absolute values of gradients which could vary between networks and datasets, matter less.

Revised: Overall, MuPPET employs the metric p as a mechanism to trigger a precision switch whenever p violates threshold T more than γ times.

Revised: The likelihood of observing 3 gradients across 3 epochs that have low gradient diversity, especially at early stages of training is low.

Revised: The intuition applied here is that when this does happen at a given precision, it may be an indication that information is being lost due to quantisation and thus corresponds to a high p value, which argues to move to a higher bitwidth.

Revised: 3.3. H YPERPARAMETERS

Original: The hyperparameters for the proposed MuPPET algorithm are the following: 1) values of α , β , and λ that deﬁne the decaying threshold from Eq.
Revised: The hyperparameters for the proposed MuPPET algorithm are the following: 1) values of α , β , and λ that deﬁne the decaying threshold from Eq.

Original: (7), 2) the number of threshold violations allowed  before the precision change is triggered ( γ ), 3) the resolution r , 4) the set of precisions at which training is performed, and 5) the epochs at which the learning rate is changed.
Revised: (8), 2) the number of threshold violations allowed [Figure] before the precision change is triggered ( γ ), 3) the resolution r , 4) the set of precisions at which training is performed, and 5) the epochs at which the learning rate is changed.

Original: The values of α , β , λ , r , and γ were set at 1, 1.5, 0.1, 3, and 2 respectively after empirical cross-validation.
Revised: The values of α , β , λ , r , and γ were set at 1, 1.5, 0.1, 3, and 2 respectively after empirical cross-validation.

Original: These were tuned by running training on AlexNet and ResNet20 on the CIFAR-10 dataset.
Revised: These were tuned by running training on AlexNet and ResNet20 on the CIFAR-10 dataset.

Original: All MuPPET hyperparameters remain the same regardless of network or dataset.
Revised: All MuPPET hyperparameters remain the same regardless of network or dataset.

Original: Regarding training hyperparameters, batch size was increased from 128 to 256 going from CIFAR-10 to ImageNet.
Revised: Regarding training hyperparameters, batch size was increased from 128 to 256 going from CIFAR-10 to ImageNet.

Original: All other training hyperparameters, mainly learning rate, weight decay and momentum, remained constant.
Revised: All other training hyperparameters, including learning rate remained constant.

Original: The remaining hyperparameters such as convolutoin type, distribution approach, loss calculation, amongst others, followed framework defaults.

Original: Analysis of generalisability and the training hyperparameters used are presented in Section 5.
Revised: Analysis of generalisability and the training hyperparameters used are presented in Section 4.1.

Original: The levels of quantised precisions at which training was performed were 8-, 12-, 14- and 16-bit ﬁxed-point.
Revised: The empirically-chosen quantised precisions at which training was performed were 8-, 12-, 14- and 16-bit ﬁxed-point.

Original: Precisions below this did not result in any progress towards convergence for any networks.
Revised: Precisions below this did not result in any progress towards convergence for any network.

Original: Overall, MuPPET introduces a policy that allows to decide at run time the best point to switch between quantisation levels.
Revised: Overall, MuPPET introduces a policy that allows to decide at run time an appropriate point to switch between quantisation levels.

Original: After training at 16-bit ﬁxed-point, the rest of the training is performed at FP32 until the desired validation accuracy is reached.
Revised: After training at 16-bit ﬁxed-point, the rest of the training is performed at FP32 until the desired validation accuracy is reached.

Original: Decaying the learning rate causes a ﬁner exploration of the optimisation space as does increasing the quantisation level.
Revised: Decaying the learning rate causes a ﬁner exploration of the optimisation space as does increasing the quantisation level.

Original: Therefore, the learning rate was kept constant during quantised training and was decayed only after switching to FP32.
Revised: Therefore, the learning rate was kept constant during quantised training and was decayed only after switching to FP32.

Original:  E VALUATION OF M U PPET 4. P RECISION S WITCHING
Revised:  E VALUATION OF M U PPET 4. G ENERALISABILITY

Original: To evaluate the ability of MuPPET to effectively choose an epoch to switch precision at, AlexNet and ResNet20 were ﬁrst trained using MuPPET on the CIFAR-100 dataset, and the epochs at which the learning switched occured was recorded.

Original: The hyperparameters for MuPPET were kept the same across all runs.

Original: Following this, AlexNet was trained using only 8-bit ﬁxed-point computations updating an FP32 master copy of weights with learning rate changes performed at the same epochs as

Original: MuPPET, and the overall number of epochs identical to that run by MuPPET. This way, the only hyperparameter that differed between these runs was the precision at which computations were performed. Using checkpoints from this 8-bit training run, the precision was switched from 8-bit to FP32 at epochs 10, 15, 25, 50, and 75.
Revised: The MuPPET framework was evaluated on its applicability across epochs, networks and datasets.

Original: The zero column remains in 8-bit ﬁxed-point for the entire duration of training.

Original: The best validation accuracy for each of these runs is shown in Table 1.

Original: The value in brackets next to the validation accuracy of the MuPPET run is the epoch at which MuPPET switched out of 8-bit into 12-bit precision.

Original: [Table] It is seen that not switching at all causes the lowest validation accuracy, hence demonstrating that a precision switching metric is necessary when training at precisions as low as 8-bit ﬁxed-point. With hindsight, the range of optimal switching epochs differ between AlexNet (10-15) and ResNet20 (25-50). For ResNet20, MuPPET performs as expected and switches out of 8-bit precision at epoch 37 which falls well within the range of optimal switching epochs seen from hindsight.
Revised: Fig.

Original: Although for AlexNet MuPPET switches later than would be expected, in doing so it manages to exploit computational gains from reduced-precision arithmetic to a greater extent while limiting damage to the validation accuracy.

Original: 4. G ENERALISABILITY

Original: The MuPPET framework was also evaluated on its applicability across epochs, networks and datasets.

Original: Figure 2 shows the value of the metric p over the epochs in blue, and the decaying threshold described in Eq.
Revised: 2 shows the value of the metric p over the epochs in blue, and the decaying threshold described in Eq.

Original: (7) in orange.
Revised: (8) in orange.

Original: The number of epochs for which training in each precision was performed is shown by the various overlay colours.
Revised: The number of epochs for which training in each precision was performed is shown by the various overlay colours.

Revised: The ﬁrst violation is denoted by a red dot and the second violation is not seen as it occurs exactly at the point of switching.

Original: The graphs show that across various networks and datasets, the values of p stay relatively similar, backing the choice of a universal decaying factor.
Revised: The graphs show that across various networks and datasets, the values of p stay relatively similar, backing the choice of a universal decaying factor.

Original: Furthermore, empirical results for CIFAR-10 indicated that changing from one ﬁxed-point precision to another too early in the training process had a negative impact on the ﬁnal validation accuracy.
Revised: Furthermore, empirical results for CIFAR-10 indicated that changing from one ﬁxed-point precision to another too early in the training process had a negative impact on the ﬁnal validation accuracy.

Original: Using a decaying threshold ensures that the value of p needs to be much higher in the initial epochs to trigger a precision change due to the volatility of p in early epochs of training.
Revised: Using a decaying threshold ensures that the value of p needs to be much higher in the initial epochs to trigger a precision change due to the volatility of p in early epochs of training.

Original: Generalisability across epochs is obtained as p accounts for the change in information relative to the maximum information available since the last precision change.

Original: Hence, the metric acknowledges the presence of temporal variations in information provided by the gradients.

Original: Generalisability across networks and datasets is maintained as p measures a ratio.

Original: Consequently, the absolute values of gradients which could vary between networks and datasets, matter less.

Original: R ESULTS 5. P ERFORMANCE E VALUATION
Revised: 4. P ERFORMANCE E VALUATION

Original: The accuracy results presented in this section utilised the proposed stochastic quantisation strategy.
Revised: The accuracy results presented in this section utilised the proposed stochastic quantisation strategy.

Original: The methodology was developed using PyTorch.
Revised: The methodology was developed using PyTorch.

Original: As the framework does not natively support low-
Revised: As the framework does not natively support lowprecision implementations, all quantisation and computations corresponding to 8-,

Original: [Figure] precision implementations, all quantisation and computations corresponding to 8-, 12-, 14-, and 16bit precisions were performed through emulation on ﬂoating-point hardware.
Revised: 12-, 14-, and 16bit precisions were performed through emulation on ﬂoating-point hardware.

Original: All hyperparameters not speciﬁed below were left as PyTorch defaults.
Revised: All hyperparameters not speciﬁed below were left as PyTorch defaults.

Original: For all networks, an SGD optimiser was used with batch sizes 128 on CIFAR-10 or 256 on ImageNet, momentum of 0.9 and weight decay of 1 e − 4 .
Revised: For all networks, an SGD optimiser was used with batch sizes 128 on CIFAR-10 or 256 on ImageNet, momentum of 0.9 and weight decay of 1 e − 4 .

Original: As a baseline, an FP32 model with identical hyperparameters (except for batch size) was trained.
Revised: As a baseline, an FP32 model with identical hyperparameters (except for batch size) was trained.

Original: The baseline FP32 training was performed by training for 150 epochs and reducing the learning rate by a factor of 10 at epochs 50 and 100.
Revised: The baseline FP32 training was performed by training for 150 epochs and reducing the learning rate by a factor of 10 at epochs 50 and 100.

Original: In order to achieve comparable ﬁnal validation accuracy to the FP32 baseline, once MuPPET triggered a precision change out of 16-bit ﬁxed-point, 45 training epochs at FP32 precision were performed.
Revised: In order to achieve comparable ﬁnal validation accuracy to the FP32 baseline, once MuPPET triggered a precision change out of 16-bit ﬁxed-point, 45 training epochs at FP32 precision were performed.

Original: The learning rate was reduced by a factor of 10 every
Revised: The learning rate was reduced by a factor of 10 every

Original: FP32 training epochs.
Revised: FP32 training epochs.

Original: For AlexNet, ResNet18, ResNet20, and GoogLeNet, the initial learning rate was set to 0.01, 0.1, 0.1, and 0.001 respectively.
Revised: For AlexNet, ResNet18, ResNet20, and GoogLeNet, the initial learning rate was set to 0.01, 0.1, 0.1, and 0.001 respectively.

Original: CIFARImageNet [Table] AlexNet 75. [Table]
Revised: The detailed breakdown of the ImageNet training runs with training and validation loss curves can be found in the Appendix A.

Original: ResNet 90.

Original: [Table]
Revised: [Table]

Original: GoogLeNet 89. [Table] [Table] Table 2 presents the achieved Top-1 validation accuracy of the proposed method and the FP32 base line, together with the accuracy difference in percentage points (pp).
Revised: Table 1 presents the achieved Top-1 validation accuracy of MuPPET and the FP32 baseline, together with the accuracy difference in percentage points (pp).

Original: As shown on the table, MuPPET is able to provide comparable Top-1 validation accuracy to standard FP32 training across both networks and datasets.
Revised: As shown on the table, MuPPET is able to provide comparable Top-1 validation accuracy to standard FP32 training across both networks and datasets.

Original: Due to a sub-optimal training setup of GoogLeNet on ImageNet, the baseline and MuPPET training severely underperformed compared to the reported state-of-the-art works.
Revised: Due to a sub-optimal training setup of GoogLeNet on ImageNet, the baseline and MuPPET training severely underperformed compared to the reported state-of-the-art works.

Original: Nevertheless, the results demonstrate the quality of training with MuPPET using identical hyperparameters.
Revised: Nevertheless, the results demonstrate the quality of training with MuPPET using identical hyperparameters.

Original: As a result, MuPPET’s performance demonstrates the effectiveness of the precision switching strategy in maintaining accuracywhile allowing for increased performance by running many epochs at lower precision.
Revised: As a result, MuPPET’s performance demonstrates the effectiveness of the precision switching strategy in achieving signiﬁcant acceleration of training time (Section 4.3) at negligible cost in accuracy by running many epochs at lower precision, particularly on very large datasets.

Original: 5. W ALL - CLOCK T IME I MPROVEMENTS
Revised: 4. W ALL - CLOCK T IME I MPROVEMENTS

Original: This section explores the gains in estimated wall-clock time of the proposed method with respect to baseline FP32 training, Mixed Precisiontraining by Micikevicius et al.
Revised: This section explores the gains in estimated wall-clock time of the current implementation of MuPPET (Current Impl.) with respect to baseline FP32 training, Mixed Precision by Micikevicius et al.

Original: (2018) and MuPPET’s theoretical bound in acceleration that would be achieved by optimising the stochasitc rounding implementation (Table 3).
Revised: (2018) and MuPPET’s ideal implementation (Table 2).

Original: For all performance results, the target platform was an NVIDIA RTX  Ti GPU.
Revised: For all performance results, the target platform was an NVIDIA RTX 2080 Ti GPU.

Original: At the moment, deep-learning frameworks, such as PyTorch, do not provide native support for reduced-precision hardware.
Revised: At the moment, deep learning frameworks, such as PyTorch, do not provide native support for reduced-precision hardware.

Original: Consequently, the wall-clock timesprovided in Table 3 were estimated using a performance model developed with NVIDIA’s CUTLASS library (Kerr et al., 2018) for reduced-precision general matrix-multiplication (GEMM) employing the latest Turing architecture GPUs.
Revised: Consequently, the wall-clock times in Table 2 were estimated using a performance model developed with NVIDIA’s CUTLASS library (Kerr et al., 2018) for reduced-precision general matrix-multiplication (GEMM) employing the latest Turing architecture GPUs.

Original: The GEMMs that were accelerated were in the convolutional and fully-connected layers of each network.
Revised: The GEMMs that were accelerated were in the convolutional and fully-connected layers of each network.

Original: INT8 hardware was used to proﬁle the 8-bit precision, while FP16 hardware was used to proﬁle
Revised: INT8 hardware was used to proﬁle the 8-bit ﬁxed-point computations, while FP16 hardware was used to proﬁle

Original: 12-, 14-, and 16-bit  computations as well as Mixed Precision (Micikevicius et al., 2018) wall-clock time.
Revised: 12-, 14-, and 16-bit ﬁxed-point computations as well as Mixed Precision (Micikevicius et al., 2018) wall-clock time.

Original: CUTLASS (Kerr et al., 2018) natively implements bit-packing to capitalise on improved memory bandwidth utilisation.
Revised: CUTLASS (Kerr et al., 2018) natively implements bit-packing to capitalise on improved memory-bandwidth utilisation.

Revised: The model for the current implementation is limited by the fact that frameworks force quantisation to happen to and from FP32.

Revised: For the MuPPET (Ideal) scenario, the model assumes native hardware utilisation which would reduce the overhead by removing this restriction.

Original: As shown on Table 3, MuPPET consistently achieves 1.25-1.32 × speedup over the FP32 baseline across the networks when targeting ImageNet on the given GPU.
Revised: As shown on Table 2, MuPPET consistently achieves 1.25-1.32 × speedup over the FP32 baseline across the networks when targeting ImageNet on the given GPU.

Original: With respect to Mixed Precision by Micikevicius et al. (2018), the proposed method outperforms it on AlexNet by 1.23 × and delivers comparable performance for ResNet18 and GoogLeNet.
Revised: With respect to Mixed Precision, the proposed method outperforms it on AlexNet by 1.23 × and delivers comparable performance for ResNet18 and GoogLeNet.

Original: Currently, the absence of native quantisation support, and hence the necessity to emulate quantisation and the associated overheads, is the limiting factor for MuPPET to achieve higher processing speed.
Revised: Currently, the absence of native quantisation support, and hence the necessity to emulate quantisation and the associated overheads, is the limiting factor for MuPPET to achieve higher processing speed.

Original: In this respect, with an optimised quantisation implementation, our method would yield 1.05 × and 1.48 × speedup for ResNet18 and 
Revised: In this respect, MuPPET run on native hardware would yield 1.05 × and 1.48 × speedup for ResNet18 and GoogLeNet respectively compared to Mixed Precision.

Original: GoogLeNet respectively over Mixed Precision.
Revised: As a result, MuPPET demonstrates consistently faster time-to-accuracy (Coleman et al., 2019) compared to Mixed Precision across the benchmarks.

Original: Additionally, performing 12-, 14- and 16-bitcomputations on precision-optimised ﬁxed-point hardware would furtheramplify MuPPET’s potential gains due to the increased parallelisation.
Revised: Additionally, while Mixed Precision has already reached its limit by using FP16 on FP16-native GPUs, the 8-, 12-, 14- and 16-bit ﬁxed-point computations enabled by MuPPET leave space for further potential speedup when targeting next- and currentgeneration (Fowers et al., 2018) precision-optimised ﬁxed-point platforms.

Original: Similar to the analysispresented in Section 5.1, Micikevicius et al.
Revised: Similar to the analysis in Section 4.2, Micikevicius et al.

Original: (2018) and Wang et al.
Revised: (2018) and Wang et al.

Original: (2018) compare theirrespective schemes to baseline FP32 training performed by them.
Revised: (2018) compare their schemes to baseline FP32 training performed by them.

Original: The reported results demonstrate that their methods achieve similar accuracy results to our method by lying close to the respective FP32 training accuracy.
Revised: The reported results demonstrate that their methods achieve similar accuracy results to our method by lying close to the respective FP32 training accuracy.

Original: As Wang et al.
Revised: As Wang et al.

Original: (2018) do not provide any results in terms of gains in wall-clock times and since they use custom FP8 hardware, it could not be directly compared to our method. 
Revised: (2018) do not provide any results in terms of gains in wall-clock times and since they use custom FP8 hardware, their work could not be directly compared to our method. FP (Baseline) Mixed Prec (Micikevicius et al., 2018) MuPPET (Current Impl.)

Revised: MuPPET (Ideal)

Original: [Table]
Revised: AlexNet 30:13 (1 × ) 29.20 (1.03 × ) 23:52 (1.27 × ) 20:25 (1.48 × ) ResNet 132:46 (1 × ) 97:25 (1.36 × ) 100:19 (1.32 × ) 92:43 (1.43 × ) GoogLeNet 152:28 (1 × ) 122:51 (1.24 × ) 122:13 (1.25 × ) 82:38 (1.84 × ) [Figure]

Revised: To evaluate the ability of MuPPET to effectively choose an epoch to switch precision at, AlexNet and ResNet were ﬁrst trained using MuPPET on the CIFAR-100 dataset.

Revised: The hyperparameters for MuPPET were kept the same across all runs.

Original: C ONCLUSION
Revised: From the results it was noted that training at reduced precision and not switching at all causes a drop in validation accuracy of 1.4% and 1.3% for AlexNet and ResNet20 respectively, hence demonstrating the need to switch precisions when training at bit-widths as low as 8-bit ﬁxed-point. To demonstrate the beneﬁts of a precision switching methodology, two further sets of experiments were conducted on ResNet20 using CIFAR100 as depicted in Fig. First, 34 training runs were performed (34 red dots in Fig. 3), where for each training four epochs along the standard training duration were randomly selected and used as the switching points.

Revised: Second, the switching strategy MuPPET generated for AlexNet and GoogLeNet was applied to ResNet20 (2 blue dots in Fig. 3).

Revised: Fig.

Revised: 3 shows the best test accuracy achieved by each of the runs and the training time as estimated by our performance model described in Sec. 4.3.

Revised: It shows that for a given time-budget, MuPPET runs (6 green dots) outperform on average all other experiment sets, demonstrating the need for a precision switching policy that is real-time and agnostic to network and dataset in order to achieve a good accuracy-to-training-time trade-off.

Revised:  C ONCLUSION

Original: This paper proposes MuPPET, a novel low-precision CNN training scheme that combines the use of ﬁxed-point and ﬂoating-point representations to produce a network trained for FP32 inference.
Revised: This paper proposes MuPPET, a novel low-precision CNN training scheme that combines the use of ﬁxed-point and ﬂoating-point representations to produce a network trained for FP32 inference.

Original: By introducing a precision-switching mechanism that decides at run time the best transition point between different precision regimes, the proposed framework achieves Top-1 validation accuracies comparable to that achieved by state-of-the-art FP32 training regimes while delivering signiﬁcant speedup in terms of training time.
Revised: By introducing a precision-switching mechanism that decides at run time an appropriate transition point between different precision regimes, the proposed framework achieves Top-1 validation accuracies comparable to that achieved by state-of-the-art FP32 training regimes while delivering signiﬁcant speedup in terms of training time.

Original: Quantitative evaluation demonstrates that MuPPET’s training strategy generalises across CNN architectures and datasets by adapting the training process to the target CNN-dataset pair during run time.
Revised: Quantitative evaluation demonstrates that MuPPET’s training strategy generalises across CNN architectures and datasets by adapting the training process to the target CNN-dataset pair during run time.

Original: Overall, MuPPET enables the utilisation of the lowprecision hardware units available on modern, specialised processors for machine learning, such as next-generation GPUs, FPGAs and TPUs,in order to yield improvement in training time and energy efﬁciency without impacting the resulting accuracy.
Revised: Overall, MuPPET enables the utilisation of the lowprecision hardware units available on modern specialised processors, such as next-generation GPUs, FPGAs and TPUs, to yield improvements in training time and energy efﬁciency without impacting the resulting accuracy.

Original: Future work will focus on the application of the proposed framework to the training of LSTMs and RNNs, where the training process is more sensitive to the quantisation of the gradients, as well as to the extension of MuPPET to includethe batch size andthe learning rate as part of its hyperparameters. 
Revised: Future work will focus on applying the proposed framework to the training of LSTMs, where the training process is more sensitive to gradient quantisation, as well as on the extension of MuPPET to include batch size and learning rate as part of its hyperparameters. Furthermore, we will explore improved quantisation techniques that could enable training convergence for bitwidths even lower than 8-bit ﬁxed-point. A A PPENDIX A

Original: Exploring the mapping of  MuPPET onto FPGA systems yields a greater performance per watt signiﬁcantly reducing the power budget and consequently operational costs.
Revised: The graphs in Fig. 4, 5 and 6 demonstrate both the training and validation loss of AlexNet, ResNet and GoogLeNet for MuPPET and FP32 runs on the ImageNet dataset. For each graph, the light gray lines indicate the point at which precision was switched in the MuPPET run. The green lines are used to show MuPPET behaviour and blue to show FP32 behaviour. Solid lines show validation loss and dashed lines show training loss.

Revised: B A PPENDIX B

Original: Finally, investigating improved quantisation techniques can provide training convergence for bitwidths even lower than 8-bit ﬁxed-point.
Revised: This section contains the larger versions of all the ﬁgures in the paper for enhanced clarity. [Figure] B e s t T e s t

Revised: A cc u r a c y ( % )

Revised: [Figure]

Revised: Resnet

Revised: [Figure]

Revised: CIFAR

Revised: [Figure]