Original: Differentiable Training for Hardware Efﬁcient LightNNs
Revised: Differentiable Training for Hardware Efﬁcient LightNNs

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: To reduce runtime and resource utilization of Deep Neural Networks (DNNs) oncustomized hardware, LightNN has been proposed by constraining the weightsof DNNs to be a sum of a limited number (denoted as k ∈ { 1 , 2 } ) of powersof 2.
Revised: To reduce runtime and resource utilization of Deep Neural Networks (DNNs) on customized hardware, LightNN has been proposed by constraining the weights of DNNs to be a sum of a limited number (denoted as k ∈ { 1 , 2 } ) of powers of 2.

Original: LightNNs can therefore replace the multiplication between activations andweights with a shift operation or two shifts and an add operation.
Revised: LightNNs can therefore replace the multiplication between activations and weights with a shift operation or two shifts and an add operation.

Original: To provide amore continuous Pareto-optimal curve of accuracy and runtime so that hardwaredesigners can have more ﬂexible options of DNN conﬁgurations, one can customizethe k for each convolutional ﬁlter.
Revised: To provide a more continuous Pareto-optimal curve of accuracy and runtime so that hardware designers can have more ﬂexible options of DNN conﬁgurations, one can customize the k for each convolutional ﬁlter.

Original: In this paper, we formulate the selection of k to bedifferentiable, and train the model weights and per-ﬁlter k in an end-to-end fashion.
Revised: In this paper, we formulate the selection of k to be differentiable, and train the model weights and per-ﬁlter k in an end-to-end fashion.

Original: Since ﬂexible- k LightNNs (FLightNNs) fully utilize the hardware resources on
Revised: Since ﬂexible- k LightNNs (FLightNNs) fully utilize the hardware resources on

Original: Field Programmable Gate Arrays (FPGAs), our experimental results show that
Revised: Field Programmable Gate Arrays (FPGAs), our experimental results show that

Original: FLightNN can achieve 2 × speedup on FPGAs when compared to LightNN-2,with only 0.1% accuracy degradation.
Revised: FLightNN can achieve 2 × speedup on FPGAs when compared to LightNN-2, with only 0.1% accuracy degradation.

Original: In addition, compared to a 4-bit ﬁxed-pointquantization, FLightNN can achieve slightly higher accuracy and 1.4 × speedup,due to its lightweight shift operations.
Revised: In addition, compared to a 4-bit ﬁxed-point quantization, FLightNN can achieve slightly higher accuracy and 1.4 × speedup, due to its lightweight shift operations.

Original:  Introduction
Revised:  Introduction

Original: Emerging vision, speech and natural language applications have widely adopted deep learning modelsand, as a result, have achieved state-of-the-art accuracy [5, 15, 3].
Revised: Emerging vision, speech and natural language applications have widely adopted deep learning models and, as a result, have achieved state-of-the-art accuracy [6, 17, 4].

Original: Furthermore, recent industrial efforthas focused on implementing the models on mobile devices [1].
Revised: Furthermore, recent industrial effort has focused on implementing the models on mobile devices [1].

Original: However, real-time applicationsbased on these deep models may incur unacceptably large latencies and can easily drain the battery onenergy-limited devices.
Revised: However, real-time applications based on these deep models may incur unacceptably large latencies and can easily drain the battery on energy-limited devices.

Original: For example, smartphones can only run the AlexNet-based object detectionfor one hour [14].
Revised: For example, smartphones can only run the AlexNet-based object detection for one hour [16].

Original: Therefore, prior research has proposed model compression techniques includingpruning and quantization to satisfy the stringent energy and latency requirements [9, 11].
Revised: Therefore, prior research has proposed model compression techniques including pruning and quantization to satisfy the stringent energy and latency requirements [10, 12, 3, 14].

Original: curve for LightNN models. As one of the recently proposed quantization approaches, LightNN [6]constrains the weights of DNNs to be a sum of k powers of 2, andtherefore can use shift and add operations to replace the multiplications between activations and weights.
Revised: As one of the recently proposed quantization approaches, LightNN [7] constrains the weights of DNNs to be a sum of k powers of 2, and therefore can use shift and add operations to replace the multiplications between activations and weights.

Original: For LightNN-1 1 , all themultiplications of the DNNs will be replaced by a shift operation,while for LightNN-2, two shifts and an add replace the multiplication.
Revised: For LightNN-1 1 , all the multiplications of the DNNs will be replaced by a shift operation, while for LightNN-2, two shifts and an add replace the multiplication.

Original: Since shift operations are much more lightweight on customizedhardware ( e.g. , FPGA or ASIC), LightNNs can achieve faster speedand lower energy consumption, and generally maintains accuracy forover-parameterized models [6].
Revised: Since shift operations are much more lightweight on customized hardware ( e.g. , FPGA or ASIC), LightNNs can achieve faster speed and lower energy consumption, and generally maintains accuracy for over-parameterized models [7].

Original: Although LightNNs provide betterenergy-efﬁciency, they lack the ﬂexibility to provide ﬁne-grained energy/delay and accuracy curve.
Revised: Although LightNNs provide better energy-efﬁciency, they lack the ﬂexibility to provide ﬁne-grained energy/delay and accuracy curve.

Original: As explained in Fig.
Revised: As explained in Fig.

Original: 1, the energyefﬁciency for these models also exhibits gaps, making the Paretooptimal curve of accuracy and energy discrete.
Revised: 1, the energy efﬁciency for these models also exhibits gaps, making the Paretooptimal curve of accuracy and energy discrete.

Original: However, a continuousaccuracy and energy/delay trade-off is an important feature for vendors to target different marketsegments ( e.g. , IoT devices, edge devices, and mobile devices).
Revised: However, a continuous accuracy and energy/delay trade-off is an important feature for vendors to target different market segments ( e.g. , IoT devices, edge devices, and mobile devices).

Original: To provide a more ﬂexible Pareto curve for the LightNN framework, we propose to equip each convolutional ﬁlter with the freedom to use a different number of shift-and-add operations to approximatemultiplications.
Revised: To provide a more ﬂexible Pareto curve for the LightNN framework, we propose to equip each convolutional ﬁlter with the freedom to use a different number of shift-and-add operations to approximate multiplications.

Original: Speciﬁcally, we introduce a set of free variables k
Revised: Speciﬁcally, we introduce a set of free variables k

Original: = { k 1 , . . .
Revised: = { k 1 , . . .

Original: , k F } where eachdimension represents the number of shift-and-add for the corresponding convolutional ﬁlter.
Revised: , k F } where each dimension represents the number of shift-and-add for the corresponding convolutional ﬁlter.

Original: As aresult, a more contiguous Pareto curve can be achieved.
Revised: As a result, a more contiguous Pareto curve can be achieved.

Original: For example, if we constrain k ∈ { 1 , 2 } F ,then the delay and energy consumption of the new model will sit between LightNN-1 ( k = { 1 } F )and LightNN-2 ( k = { 2 } F ).
Revised: For example, if we constrain k ∈ { 1 , 2 } F , then the delay and energy consumption of the new model will sit between LightNN-1 ( k = { 1 } F ) and LightNN-2 ( k = { 2 } F ).

Original: Formally, we are solving min w , k L ( w , k ) , where L is the loss functionand w is the weights.
Revised: Formally, we are solving min w , k L ( w , k ) , where L is the loss function and w is the weights.

Original: However, the commonly adopted stochastic gradient descent (SGD) does notapply in this case since L is non-differentiable w.r.t.
Revised: However, the commonly adopted stochastic gradient descent (SGD) does not apply in this case since L is non-differentiable w.r.t.

Original: k .
Revised: k .

Original: In this paper, we propose a differentiabletraining algorithm which enables end-to-end optimization with standard SGD.
Revised: In this paper, we propose a differentiable training algorithm which enables end-to-end optimization with standard SGD.

Original: The resulting networkis dubbed F LightNN for its ﬂexible k values.
Revised: The resulting network is dubbed F LightNN for its ﬂexible k values.

Original:  Differentiable Training for FLightNN
Revised:  Differentiable Training for FLightNN

Original: In this section, we ﬁrst deﬁne the quantization function, and then introduce the end-to-end trainingalgorithm for FLightNN, equipped with a regularization loss to penalize large k values.
Revised: In this section, we ﬁrst deﬁne the quantization function, and then introduce the end-to-end training algorithm for FLightNN, equipped with a regularization loss to penalize large k values.

Original: 2.1 Quantization function
Revised: 2.1 Quantization function

Original: We ﬁrst denote the i th ﬁlter of the network as w i and the quantization function for the ﬁlter w i as
Revised: We ﬁrst denote the i th ﬁlter of the network as w i and the quantization function for the ﬁlter w i as

Original: Q k ( w i | t ) , where k = max i k is the maximum number of shifts used for this network, and vector tis a latent variable that controls the approximation ( e.g. , some threshold values).
Revised: Q k ( w i | t ) , where k = max i k is the maximum number of shifts used for this network, and vector t is a latent variable that controls the approximation ( e.g. , some threshold values).

Original: Also, we denote theresidual resulting from the approximation as r i,k
Revised: Also, we denote the residual resulting from the approximation as r i,k

Original: = w i − Q k ( w i | t ) .
Revised: = w i − Q k ( w i | t ) .

Original: Then, we formally deﬁne thequantization function as follows:
Revised: Then, we formally deﬁne the quantization function as follows:

Original: [Equation]
Revised: [Equation]

Original: where R ( x ) = sign ( x ) × 2 [ log ( | x | )] rounds the input variable to a nearest power of 2, and [ . ] is arounding-to-integer function.
Revised: where R ( x ) = sign ( x ) × 2 [ log ( | x | )] rounds the input variable to a nearest power of 2, and [ . ] is a rounding-to-integer function.

Original: To interpret the thresholds t , t 0 determines whether this ﬁlter is prunedout, and t 1 determines whether one shift is enough, etc.
Revised: To interpret the thresholds t , t 0 determines whether this ﬁlter is pruned out, and t 1 determines whether one shift is enough, etc.

Original: Then, the number of shifts for the i -th ﬁlteris k i
Revised: Then, the number of shifts for the i -th ﬁlter is k i

Original: = (cid:80) k − 1 j =0 1 ( || r i,j || 2 > t j ) .
Revised: = (cid:80) k − 1 j =0 1 ( || r i,j || 2 > t j ) .

Original: Therefore, choosing k i per ﬁlter is equivalent to ﬁnding optimalthresholds t .
Revised: Therefore, choosing k i per ﬁlter is equivalent to ﬁnding optimal thresholds t .

Original: 2.2 Differentiable training
Revised: 2.2 Differentiable training

Original: Instead of picking the threshods t by hand, we consider them as trainable parameters.
Revised: Instead of picking the threshods t by hand, we consider them as trainable parameters.

Original: Therefore, theloss function L ( w , t ) is a function of both weights and thresholds.
Revised: Therefore, the loss function L ( w , t ) is a function of both weights and thresholds.

Original: Similar to prior work on DNNquantization [17, 4], we use the straight-through estimator (STE) [2] to compute ∂ L ∂ w i .
Revised: Similar to prior work on DNN quantization [19, 5], we use the straight-through estimator (STE) [2] to compute ∂ L ∂ w i .

Original: By deﬁning∂ w qi ∂ w i = 1 where w qi =
Revised: By deﬁning ∂ w qi ∂ w i = 1 where w qi =

Original: Q k ( w i | t ) is the quantized w i ; therefore, we have ∂ L ∂ w i =
Revised: Q k ( w i | t ) is the quantized w i ; therefore, we have ∂ L ∂ w i =

Original: ∂
Revised: ∂

Original: L ∂ w qi · ∂ w qi ∂ w i =
Revised: L ∂ w qi · ∂ w qi ∂ w i =

Original: ∂
Revised: ∂

Original: L ∂ w qi ,which becomes a differentiable expression.
Revised: L ∂ w qi , which becomes a differentiable expression.

Original: To compute the gradient for thresholds, i.e. , ∂ w qi ∂ t j , we relax the indicator function g ( x, t j )
Revised: To compute the gradient for thresholds, i.e. , ∂ w qi ∂ t j , we relax the indicator function g ( x, t j )

Original: = 1 ( x > t j )to a sigmoid function [8], σ ( . ) , when computing gradients, i.e. , ˆ g ( x, t j ) =
Revised: = 1 ( x > t j ) to a sigmoid function [9], σ ( . ) , when computing gradients, i.e. , ˆ g ( x, t j ) =

Original: σ ( x − t j ) .
Revised: σ ( x − t j ) .

Original: In addition,we use STE to compute the gradient for R ( x ) .
Revised: In addition, we use STE to compute the gradient for R ( x ) .

Original: Thus, the gradient ∂ w qi ∂ t j can be computed by:
Revised: Thus, the gradient ∂ w qi ∂ t j can be computed by:

Original: [Equation]
Revised: [Equation]

Original: where ∂ || r i,l || 2 ∂ t j and ∂ r i,l ∂ t j are 0 for l < j ; otherwise, they can be computed with the result of ∂ Q l ( w i | t ) ∂ t j .∂ t l ∂ t j = 1 ( l = j ) .
Revised: where ∂ || r i,l || 2 ∂ t j and ∂ r i,l ∂ t j are 0 for l < j ; otherwise, they can be computed with the result of ∂ Q l ( w i | t ) ∂ t j . ∂ t l ∂ t j = 1 ( l = j ) .

Original: 2.3 Regularization
Revised: 2.3 Regularization

Original: To encourage smaller k i for the ﬁlters, we also add regularization loss:
Revised: To encourage smaller k i for the ﬁlters, we also add regularization loss:

Original: L reg,k ( w ) =77 (cid:80) k −j =

Revised: L reg,k ( w ) = (cid:80) k −j =

Original: λ j (cid:80)
Revised: λ j (cid:80)

Original: i ||
Revised: i ||

Original: r
Revised: r

Original: i,j || 2 where λ j performs as a handle to balance accuracy and model sparsity.
Revised: i,j || 2 where λ j performs as a handle to balance accuracy and model sparsity.

Original: Thisregularization loss is the sum of several group Lasso losses, since they can introduce structuralsparsity [13].
Revised: This regularization loss is the sum of several group Lasso losses, since they can introduce structural sparsity [15].

Original: The ﬁrst item λ 0 (cid:80)
Revised: The ﬁrst item λ 0 (cid:80)

Original: i ||
Revised: i ||

Original: r i, 0 ||
Revised: r i, 0 ||

Original: 2 = λ 0 (cid:80)
Revised: 2 = λ 0 (cid:80)

Original: i ||
Revised: i ||

Original: w i
Revised: w i

Original: || 2 is used to prune the whole ﬁlters out,while the other items ( j > 0 ) regularize the residuals.
Revised: || 2 is used to prune the whole ﬁlters out, while the other items ( j > 0 ) regularize the residuals.

Original:  Experimental Results
Revised:  Experimental Results

Original: [Table]

Original: We conduct experiments on both large and small CNNs for CIFAR-10 datasets.
Revised: We conduct experiments on both large and small CNNs for CIFAR-10 datasets.

Original: The conﬁguration for
Revised: The conﬁguration for

Original: Network-1 is: (2 × 128 C 3) − MP 2 − (2 × 256 C 3) − MP 2 − (2 × 512 C 3) − 10 C 3 − GP , where2 × 128 C 3 means two convolutional layers each with 128 3 × 3 ﬁlters; MP2 means max pooling witha 2x2 kernel; GP means global pooling.
Revised: Network-1 is: (2 × 128 C 3) − MP 2 − (2 × 256 C 3) − MP 2 − (2 × 512 C 3) − 10 C 3 − GP , where 2 × 128 C 3 means two convolutional layers each with 128 3 × 3 ﬁlters; MP2 means max pooling with a 2x2 kernel; GP means global pooling.

Original: Network-2 reduces the number of ﬁlters per convolutionallayer (except the last layer) of Network-1 by a factor of 8.
Revised: Network-2 reduces the number of ﬁlters per convolutional layer (except the last layer) of Network-1 by a factor of 8.

Original: For both networks, each convolutionallayer is followed by a batch normalization layer and a Leaky ReLU activation function [12].
Revised: For both networks, each convolutional layer is followed by a batch normalization layer and a Leaky ReLU activation function [13].

Original: We usethe Adam optimizer [10] to train the network.
Revised: We use the Adam optimizer [11] to train the network.

Original: The learning rate starts at 5e-3, and decrease by 0.1every 80 epochs.
Revised: The learning rate starts at 5e-3, and decrease by 0. every 80 epochs.

Original: All the models are trained for 200 epochs.
Revised: All the models are trained for 200 epochs.

Original: For FLightNN, we initialize the threshodst to 0 , and set the largest shifts k as 2.
Revised: For FLightNN, we initialize the threshods t to 0 , and set the largest shifts k as 2.

Original: For all, except the 32-bit full-precision model, we use 8-bitﬁxed-point quantization for the activations.
Revised: For all, except the 32-bit full-precision model, we use 8-bit ﬁxed-point quantization for the activations.

Original: By varying λ , we can have different accuracy-runtimetrade-offs for FLightNN.
Revised: By varying λ , we can have different accuracy-runtime trade-offs for FLightNN.

Original: The results are shown in Table 1, where for each network we train two
Revised: The results are shown in Table 1, where for each network we train two

Original: FLightNNs with different ˘ .
Revised: FLightNNs with different ˘ .

Original: We also implement and synthesize the models on Xilinx ZC706 FPGA,and measure the runtime of batched inference.
Revised: We also implement and synthesize the models on Xilinx ZC706 FPGA, and measure the runtime of batched inference.

Original: In Table 1, the per-image runtime and speedup for thelargest convolutional layer is reported.
Revised: In Table 1, the per-image runtime and speedup for the largest convolutional layer is reported.

Original: The accuracy-runtime
Revised: The accuracy-runtime

Original: -offs are also shown in Fig.
Revised: -offs are also shown in Fig.

Original: [Table]
Revised: [Table]

Original: 1 ×

Original: LightNN-2 8 W 8 A 92.
Revised: -2 8 W 8 A 92.

Original: 7.8 ×
Revised: 7.8 ×

Original: LightNN-1 4 W 8 A 91.
Revised: LightNN-1 4 W 8 A 91.

Original: 30.2 ×
Revised: 30.2 ×

Original: Fixed-point 4 W 8 A 92.
Revised: Fixed-point 4 W 8 A 92.

Original: 15.2 ×
Revised: 15.2 ×

Original: FLightNN92.
Revised: FLightNN92.

Original: 30.2 ×
Revised: 30.2 ×

Original: FLightNN92.
Revised: FLightNN92.

Original: 21.0 ×  0.08M
Revised: 21.0 × [Figure] 0.08M

Original: Full-precision 86.
Revised: Full-precision 86.

Original: 1 ×
Revised: 1 ×

Original: LightNN-2 8 W 8 A 86.
Revised: LightNN-2 8 W 8 A 86.

Original: 7.0 ×
Revised: 7.0 ×

Original: LightNN-1 4 W 8 A 84.
Revised: LightNN-1 4 W 8 A 84.

Original: 14.4 ×
Revised: 14.4 ×

Original: Fixed-point 4 W 8 A 85.
Revised: Fixed-point 4 W 8 A 85.

Original: 10.5 ×
Revised: 10.5 ×

Original: FLightNN85.
Revised: FLightNN85.

Original: 15.0 ×
Revised: 15.0 ×

Original: FLightNN85.
Revised: FLightNN85.

Original: 12.6 ×
Revised: 12.6 ×

Original: [Figure]
Revised: [Figure]

Original: As shown in Table 1 and Fig.
Revised: As shown in Table 1 and Fig.

Original: 2, FLightNN shows the advantage of ﬂexible accuracy-speed tradeoffs.
Revised: 2, FLightNN shows the advantage of ﬂexible accuracy-speed tradeoffs.

Original: For both networks, FLightNNs can achieve the accuracy close to LightNN-2, but have muchhigher speedup.
Revised: For both networks, FLightNNs can achieve the accuracy close to LightNN-2, but have much higher speedup.

Original: Also, FlightNNs have higher accuracy and a lower speedup than LightNN-1.
Revised: Also, FlightNNs have higher accuracy and a lower speedup than LightNN-1.

Original: Thus,
Revised: Thus,

Original: FLightNNs provide continuous trade-off curves for accuracy and speed.
Revised: FLightNNs provide continuous trade-off curves for accuracy and speed.

Original: Compared to the ﬁxed-pointquantization, FLightNN can achieve higher accuracy, and up to 1 .
Revised: Compared to the ﬁxed-point quantization, FLightNN can achieve higher accuracy, and up to 1 .

Original: 4 × speedup.
Revised: 4 × speedup.

Original: This is because themultiplication is replaced by shift operators, which require much fewer resources on FPGA thanmultipliers.
Revised: This is because the multiplication is replaced by shift operators, which require much fewer resources on FPGA than multipliers.

Original: Therefore, the computation for FLightNN can be unrolled more times than that ofﬁxed-point DNNs.
Revised: Therefore, the computation for FLightNN can be unrolled more times than that of ﬁxed-point DNNs.

Original: By comparing the ﬁrst FLightNN-2 of Network-2 with LightNN-1, we ﬁnd that
Revised: By comparing the ﬁrst FLightNN-2 of Network-2 with LightNN-1, we ﬁnd that

Original: FLightNN-2 can achieve higher accuracy even with the same storage as LightNN-1.
Revised: FLightNN-2 can achieve higher accuracy even with the same storage as LightNN-1.

Original: This is becauseinitially FLightNN-2 quantizes all the ﬁlters with two shifts (since t is initialized as 0 ), and graduallyadds constraints to the ﬁlters.
Revised: This is because initially FLightNN-2 quantizes all the ﬁlters with two shifts (since t is initialized as 0 ), and gradually adds constraints to the ﬁlters.

Original: This gradual quantization may be better than training a network withonly one shift from scratch, as LightNN-1 does.
Revised: This gradual quantization may be better than training a network with only one shift from scratch, as LightNN-1 does.

Original: The beneﬁt of gradual quantization has also beenobserved by prior work [16, 7] which shows that gradually imposing quantization constraints canachieve better accuracy than directly quantizing with a strict constraint.
Revised: The beneﬁt of gradual quantization has also been observed by prior work [18, 8] which shows that gradually imposing quantization constraints can achieve better accuracy than directly quantizing with a strict constraint.