From: NIPS.cc/2018/Workshop/CDNNRIA/Paper50/Authors
comment: We would like to thank the reviewer for his/her comments.

1- We combined the conclusion and experimental results due to a lack of space. We would have liked to further elaborate on our results, and we apologize that the paper appears incomplete.

2- Furthermore, please note that  the idea of the EDP has only been used for fixed-point numbers within a neural network and we are the first to use EDP for floating point, posit, and fixed-point, and compare performance. We would like to clarify that the EDP operation is non-standard and necessitated a detailed definition of all different number system, even though it is not defined by the IEEE 754 standard; the conversion of the products of each of the three numerical formats to a wide fixed-point representation yields the exact result of a MAC operation before rounding. Normally, this operation is inexact by using fused multiply-add in floating point number system with propagated error through each accumulation. Additionally, we could not find the reference you mentioned. Could you please provide us with the actual reference? Thank you.

3- We performed experiments for various exponent sizes in floating, integer and fraction sizes in fixed-point, and es sizes for posit. Therefore, for each dataset, we have multiple runs, about 44 runs of varying configurations to find the optimal parameters for each precision. Furthermore, the EDP implementation of the floating point and EDP implementation of the posit hardware are emulated in the software and therefore running each configuration is slow. However, to show the performance of these different number systems for CIFAR-10, we selected a new run that encompasses 100 randomly selected samples from CIFAR10, that can complete within the 3 day revision process. When we compare the accuracies across the different bit-width precisions and numerical formats, the results match the previous two full datasets; posit consistently shows better performance at lower bits and comparable performance to a float 32-bit. 

4- The PositNN is the name of our framework for evaluating the floating point, fixed-point, and posit low-precision numerical formats. We agree that the novelty is not in the network structure, rather in the adoption of posit for any given deep network using EDP. The idea of EDP has only been used for fixed-point numbers within a neural network. This is the first time EDP is used for floating point, posit, and fixed-point, and a comparative analysis is performed on these networks. These comparisons are valuable to the community in identifying a suitable numerical format. We empirically show better performance for the posit number system, specifically the performance of a 5-bit posit is generally unknown and we provide preliminary results indicating its candidacy in such a role.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper50/Authors
comment: We would like to thank the reviewer for his/her comments.

1- We believe posit outperforms fixed-point and floating point per the theoretical and experimental benefits from "Beating floating point at its own game: Posit arithmetic, Supercomputing Frontiers and Innovations" (J. L. Gustafson and I. T. Yonemoto); posits have fewer redundant representations, higher accuracy, and more flexible dynamic range over floating point. The redundant representations in floating point (i.e. the infinities, zeros, and NaNs) become especially cumbersome at lower precision.  Furthermore, From our point of view, the tapered precision is explained why posit performance is better. This is similar to a dynamic fixed point (Flexpoint) [1,2]. The dynamic fixed point performance is better than fixed point performance since the dynamic range and precision can be changed based on DNN parameters by using the scaling factor. However, the posit naturally has this characteristic without using the scaling factor. Therefore, the quantization error reduces significantly in the posit number system.

2- We performed experiments for various exponent sizes in floating, integer and fraction sizes in fixed-point, and es sizes for posit. Therefore, for each dataset, we have multiple runs, about 44 runs of varying configurations to find the optimal parameters for each precision. Furthermore, the EDP implementation of the floating point and EDP implementation of the posit hardware are emulated in the software and therefore running each configuration is slow. However, to show the performance of these different number systems for CIFAR-10, we selected a new run that encompasses 100 randomly selected samples from CIFAR10, that can complete within the 3 day revision process. When we compare the accuracies across the different bit-width precisions and numerical formats, the results match the previous two full datasets; posit consistently shows better performance at lower bits and comparable performance to a float 32-bit. 

3- The posit number system is a new number system. While posit has not been as widely adopted as floating point or fixed-point in hardware, there is growing interest in posit hardware (in the open-source RISC-V architecture and several papers [3,4]), and posit hardware units are simpler than those of IEEE floating-point. Furthermore, there are several startups and companies are designing Posit hardware and software for their applications. The survey of these efforts can be found on this link ( https://posithub.org/docs/PDS/PositEffortsSurvey.html). Additionally, there is a strong community to develop a standard for Posit arithmetic and we believe that the posit arithmetic will be used in many applications of AI.

[1] Gysel, Philipp, et al. "Ristretto: A framework for empirical study of resource-efficient inference in convolutional neural networks." IEEE Transactions on Neural Networks and Learning Systems (2018).

[2] Köster, Urs, et al. "Flexpoint: An adaptive numerical format for efficient training of deep neural networks." Advances in Neural Information Processing Systems. 2017.

[3] Jaiswal, Manish Kumar, and Hayden K-H. So. "Universal number posit arithmetic generator on FPGA." Design, Automation & Test in Europe Conference & Exhibition (DATE), 2018. IEEE, 2018.

[4] Podobas, Artur, and Satoshi Matsuoka. "Hardware Implementation of POSITs and Their Application in FPGAs." 2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW). IEEE, 2018.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper50/Authors
comment: We would like to thank the reviewer for his/her comments.

For implementation, we designed a customized neural network in Keras which can work with the three number systems. The EDP for arbitrary precision fixed-point accumulation is implemented in C++ with TensorFlow Op kernels to make such functions available to the Keras model. We believe the hardware-constrained EDP was nontrivial, especially with the efficient emulation of each format to mimic low-precision operations. Furthermore, please note that  the idea of the EDP has only been used for fixed-point numbers within a neural network and we are the first to use EDP for floating point, posit, and fixed-point, and compare performance. Our contribution includes this novel comparison and implementation, not just the combination of posit and a neural network. The comparisons are valuable to the community in selecting a suitable number system and we empirically show better performance of the posit number system.

We demonstrate that the tapered precision explains posit’s performance, which is similar to dynamic fixed point (Flexpoint) [1,2]. The Flexpoint performance is better than fixed-point performance since the dynamic range and precision can be changed based on DNN parameters by using a scaling factor. However, the posit naturally has this characteristic without using such a scaling factor. Therefore, the quantization error reduces significantly in the posit number system.

1- The information theory aspect of the posit number system is analyzed in reference [3]. The exponent part of the posit number system is encoded as Golomb rice coding and it is categorized as universal encoding since the numbers have unique representations, the number of bits needed to represent a value do not increase monotonically, and the length of a coding is the constant factor of length of binary encoding. Furthermore, these types of coding (Golomb rice) for the posit number system introduce the tapered precision characteristics as shown in Figure 2 in reference [3], which increases the accuracy of a posit in the small dynamic range. The suggestion for finding the bound of encoded information and Shannon maximum entropy is aptly captured and we are infact working on this. 

2- We performed experiments for various exponent sizes in floating, integer and fraction sizes in fixed-point, and es sizes for posit. Therefore, for each dataset, we have multiple runs, about 44 runs of varying configurations to find the optimal parameters for each precision. Furthermore, the EDP implementation of the floating point and EDP implementation of the posit hardware are emulated in the software and therefore running each configuration is slow. However, to show the performance of these different number systems for CIFAR-10, we selected a new run that encompasses 100 randomly selected samples from CIFAR10, that can complete within the 3 day revision process. When we compare the accuracies across the different bit-width precisions and numerical formats, the results match the previous two full datasets; posit consistently shows better performance at lower bits and comparable performance to a float 32-bit. 

[1] Gysel, Philipp, et al. "Ristretto: A framework for empirical study of resource-efficient inference in convolutional neural networks." IEEE Transactions on Neural Networks and Learning Systems (2018).

[2] Köster, Urs, et al. "Flexpoint: An adaptive numerical format for efficient training of deep neural networks." Advances in Neural Information Processing Systems. 2017.

[3] Lindstrom, Peter, Scott Lloyd, and Jeffrey Hittinger. "Universal coding of the reals: alternatives to IEEE floating point." Proceedings of the Conference for Next Generation Arithmetic. ACM, 2018.

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper50/AnonReviewer2
title: Additional experiments needed to pass the NIPS theshold
review: Summary:

This paper proposes the use of posit numerical format in NN architectures. It motivates this choice of format by comparing a histogram of value range for posit with that observed for a conv net in practice and shows how posit is well suited because it allocates more precision to values between -1 and 1 which tend to occur most in practice as per the histogram comparison. After introducing the posit format, the authors define a tapered precision NN architecture which includes a quantize and clip module to show how values are mapped to posit numerical format and a EDP module to minimize arithmetic error during the multiply accumulate computation, by holding off on rounding till the end of the operation. Performance results are shown on MNIST and Fashion MNIST, comparing posit, floating, and fixed point representations at different bit levels (i.e. 5-8 bits).

Pros:

Seeing as the posit format already exists and the idea of reducing the number of bits in NNs already exists, the main contribution is how to combine posit with a NN architecture as summarized in Figure 2, along with the EDP module. I find the steps required for this combination to be fairly trivial, for instance the rounding and clipping operations are common in practice and the notation of holding off rounding until the end of the multiply-accumulate operation is common in many engineering implementations. The bigger advancement to me comes from the results, specifically how there is significant increase in performance at a low number of bits (i.e. 5-6 bits) compared to fixed and floating point representation. Despite the fact that the implementation aspects of posit in NN architectures is fairly trivial, this by no way diminishes this noticeable performance gain mentioned which I consider to be a meaningful contribution.


Cons:

Given that the implementation is fairly straightforward I would have liked to see the authors spend more time on additional experiments and more theoretical justification to make the paper more thorough.

1)Specifically with respect to theoretical justification, it would have been nice to see the authors justify the usage of posit format from an information theory standpoint (in terms of giving a bound on the amount of information that can be contained specifically when using this format in a NN architecture vs. floating or fixed point) in addition to the empirical observation they had with the histogram comparison in figure 1.

2)With respect to additional experiments, I would have liked to see experiments on larger datasets to show that these improvements remain the same or better at different scales. In addition to larger datasets, I would have liked to see different tasks than images to see how the framework performs on different modalities.


Recommendation:

If the authors had expanded on the above two points, I would have rated this paper a weak accept despite the somewhat trivial implementation but because they haven't I rate it as a weak reject.
rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper50/AnonReviewer1
title: Not novel enough, no reasonable-scale experiments
review: This paper proposes to use a new standard (Posit arithmetic) for representing neural network weights. Some techniques such as EDP is used to minimize arithmetic error. Experiments on MNIST and Fashion-MNIST show posit outperforms floating point and fixed point representations. 

Pros:
1. The fact that posit can outperform float and fixed by a large margin, when restricted to 5/6-bit, is interesting.
2. The method is straightforward, easy to understand, and easy to implement.

Cons:
1. The paper lacks explanation on why posit is better, which I think is important. Probably a more detailed explanation on posit can help readers understand more about this new standard.
2. Experiments are only conducted on MNIST and FASHION-MNIST, which are too small-scale and not convincing. At least CIFAR-10 dataset can be considered, to demonstrate the advantages of positNN.
3. The paper seems to only use the posit standard directly, without other proposed novel techniques. In addition, Posit is not a very popular standard compared with float or fixed, and I doubt the applicability of PositNN, e.g., to GPUs. This may be the reason why the paper only runs experiments on (FASHION-)MNIST.
rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/CDNNRIA/Paper50/AnonReviewer3
title: Review comments 
review: This paper proposes a new neural network which based on the following research: "J. L. Gustafson and I. T. Yonemoto: Beating floating point at its own game: Posit arithmetic, Supercomputing Frontiers and Innovations." Moreover, this neural network's change regular Numerical Format, i.e., float32 to Posit Numerical Format, this article shows a new aspect of solving IoT and smart wearables, they propose a new neural network architecture which has some delightful experimental result.

Pros:
1. Technically sound.
2. Easy to read and implement.

Cons:
1. The structure of the article is not complete, and the conclusion part is missing.  

2. In the neural-network (NN) architecture section, the structure of NN and implementation of NN is mixed together and there too much description of algorithms, which are already used, i.e., "MAC operation," "EDP." To give some proof or explanation of the paper propose layer (Quantize and Clip) such as Kaiming He et al. (2016) can make the article more convincing.  

3. Moreover, the experience part MNIST & Fashion MNIST is not persuasive, nevertheless as author state that model they purpose is "for the edge" which has a large amount of usage, i.e., computer vision (CV), so test more dataset as CiFar100 or ImageNet would make the experiment more convincing.  

4. Showing more innovative point in this Posit-NN is a critical problem this paper should improve in the future. From figure2 it seems this Posit-NN merely combine a hardware-friendly format which called Posit to replace float32 format and other algorithms merely quote by others work like EDP and MAC operation. Moreover, the structure is just like CNN is hard to find out what innovation of this paper propose. Thus, this Posit neural network seems just like an application of posit Numerical format not have enough innovative points.  


rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct