PositNN: Tapered Precision Deep Learning Inference for the Edge
Anonymous Author(s)
AfﬁliationAddress email
Abstract
The performance of neural networks, especially the currently popular form of deepneural networks, is often limited by the underlying hardware.
Computations indeep neural networks are expensive, have large memory footprint, and are powerhungry.
Conventional reduced-precision numerical formats, such as ﬁxed-pointand ﬂoating point, cannot accurately represent deep neural network parameterswith a nonlinear distribution and small dynamic range.
Recently proposed positnumerical format with tapered precision represents small values more accuratelythan the other formats.
In this work, we propose an ultra-low precision deep neuralnetwork, PositNN, that uses posits during inference.
The efﬁcacy of PositNN isdemonstrated on a deep neural network architecture with three datasets (MNIST, Fashion MNIST and Cifar-10), where an 8-bit PositNN outperforms other {5-8}-bitlow-precision neural networks and a 32-bit ﬂoating point baseline network.
Introduction
Hierarchical representation learning [1] frameworks such as deep neural networks (DNNs) haveachieved state-of-the-art accuracy in a wide spectrum of applications, such as computer vision [2],medical applications [3], and natural language processing [4].
Typically, DNN inference requiresmillions of parameters and billions of ﬂoating point operations ( e.g. ,
≈ 12 . 5 billion FLOPs for
DenseNet [2] with 264 layers).
DNNs are pushing towards memory bandwidth of ≈ 900GB/sfor training and ≈ 400GB/s for inference, pushing designers to choose high-bandwidth memory.
Implementing these high-bandwidth, memory intensive DNN operations on the edge, such as IoTand smart wearables, is progressively onerous due to the limited compute and memory availableon-device [5, 6].
Applications that impose near instantaneous inference, demand decentralizationfor improved security or reduced bandwidth on servers, or customization on the edge.
In several ofthese application scenarios, deploying conventional DNNs on the end device is prohibitive becauseof the data movement cost, long latencies, and increased power dissipation.
Recent studies addressthis problem using new compute and memory efﬁcient DNN architectures, parameter efﬁcient neural networks [7, 8], pruning and truncation, distillation and low-precision arithmetic [9].
Amongst the techniques to compress neural network parameters, low-precision arithmetic has been most successfulin reducing latency, memory requirements, and power consumption, especially for DNN inference[10, 9].
Linear and nonlinear quantization are the common approaches in low-precision arithmetic [11, 9].
However, the accuracy of inference is degraded when the DNN parameters are quantized to ultra-low bit-precision ≤ 8-bit [12].
To recover this accuracy degradation, the network can be retrained or the number of parameters can be signiﬁcantly increased [11, 13].
This results in increased computational complexity, as DNN training requires ≈ 3 × more computation compared to DNN inference [14].
Reducing the precision of learned parameters directly to a lower precision numerical format (ﬁxed- point, ﬂoating point, posit [15]) has shown to mitigate the overhead of quantization and retraining
0.4 Parameter Value
D e n s i t y
In this paper, we present ultra low precision deep neural networks using posit numerical format(PositNN).
The advantage of this network is that the posit numbers are represented in a nonlineartapered precision similar to the DNN inference parameters (shown in Fig. 1).
Furthermore, weaddress the accuracy degradation of rounding by using exact dot product algorithm for multiply andaccumulate operations.
The key concept is to postpone the rounding operation to the accumulationphase.
Posit Numerical Format
The posit numerical format, a Type III unum, is a tapered accuracy numerical format that representsreal numbers [15].
Numbers with a smaller exponent are represented more accurately in comparisonto the numbers with large absolute exponents since the exponent has approximately a Gaussiandistribution.
The value of a posit number is represented by Equation 1, where s represents the sign, es and fsrepresent the maximum number of bits allocated for the exponent and fraction, respectively, e and findicate the exponent and fraction values, respectively, and k , as computed by Equation 2, representsthe regime value.
[Equation]
The regime bit ﬁeld is encoded based on the runlength m of identical bits ( r...r ) terminated byeither a regime terminating bit r or the end of the n -bit value.
Note that there is no requirement todistinguish between negative and positive zero since only a single bit pattern ( 00 ... 0 ) represents zero.
Furthermore, instead of deﬁning a NaN for exceptional values and inﬁnity by various bit patterns,a single bit pattern ( 10 ... 0 ) , "Not-a-Real" ( NaR ), represents exception values and inﬁnity.
Moredetails about the posit number format can be found in [15].
[Equation]
PositNN Architecture
The PositNN architecture is shown in Fig.
2 wherein each feature F i of the convolutional layersis extracted by F i
= B i + (cid:80) C
× R × S i =
A i × W i where B i indicates the bias term, W i is the weightsmatrix,
A i represents the activation, and ( C, R, S ) are ﬁlter parameters: the number of ﬁlter channels,the ﬁlter height, and the number of ﬁlter weights, respectively.
The extracted features are used forclassiﬁcation, computed by Y =
B i + (cid:80) Ni
=
A i × W i where Y and N represents the number ofnodes and the number of outputs in each fully connected layer, respectively.
Based on these equations,the fundamental computation in the convolutional and fully connected layers is the MAC operation.
In this work, as shown in Fig. 2, the MAC operations of the convolutional and fully connected layersin the DNN are customized.
Speciﬁcally, the MAC operation, which is commonly performed by theinexact fused-multiply accumulation algorithm, is calculated by using the Exact Dot Product (EDP)algorithm where the rounding operation within MAC operations is postponed until every product hasbeen accumulated, which minimizes the MAC arithmetic error.
To perform the EDP, ﬁrstly, the 32-bit high-precision ﬂoating point weights and activations arequantized to the ≤ 8-bit low-precision posit weights and activations.
To manage the overﬂow andunderﬂow during conversion, the high-precision values that lie outside posit dynamic range areclipped (clip ( x ) ) to either the format’s maximum or minimum.
The unexceptional values duringconversion are rounded to the nearest number ( φ ( · ) ) that can be represented in the desired positnumber system.
To compute the products, the posit weights and activations are multiplied in aposit format without rounding or truncation at the end of multiplications to preserve precision.
Toavoid rounding during accumulation, the products are stored in a wide register with a width of
Q r = 2 es +2 × ( n − 2) + 2 + (cid:100) log 2 ( N op ) (cid:101) .
The products are then converted to ﬁxed-point format
FX ( m k ,n k ) , where m k = 2 es +1 × ( n − 2) + 2 + (cid:100) log 2 ( N op ) (cid:101) and n k = 2 es +1 × ( n − 2) .
Finally,the N op ﬁxed-point products are accumulated, and the result is converted back to the posit format.
[Figure]
Experimental Results
The accuracy of PositNN for the two benchmark tasks on 32-bit ﬂoating point and for {5-8}-bit lowprecision formats are shown in Table 1 and 2.
All the networks are implemented using Keras [17]and TensorFlow [18].
Universal library [19] is used for posit and ﬂoating point number exact dotproduct operation.
PositNN with 8-bit ( es = 2 ) precision outperformed {5-8}-bit ﬂoating point,ﬁxed point and 32-bit ﬂoating point numerical format.
Furthermore, when comparing 5-bit precisionnetworks, PositNN demonstrates 6.67% improvement over 5-bit ﬂoating-point on MNIST dataset.
We hypothesize that PositNN has better accuracy at lower-bit precisions, as the non-linear distributionof posit numbers is similar to the DNN inference parameters unlike in ﬁxed or ﬂoat.
This holds true,especially in the range [ − 4 , 4] , where most of the DNN inference parameters take place.
For 1%accuracy degradation in PositNN (5-bit), we estimate that there is 84.4% reduction in memory storagefor the respective inference parameters when compared to a 32-bit ﬂoating point numerical format
[Table] [Table]
PositNN accuracy on two datasets with {5-8}-bit precision compared to ﬁxed and ﬂoat (Respective best results are when posit has es ∈ { 0 , 1 , 2 } and ﬂoating point w e ∈ { 3 , 4 } )
[Table]