Paper:
Parsimonious Bayesian deep networks
Combining Bayesian nonparametrics and a forward model selection strategy, we construct parsimonious Bayesian deep networks (PBDNs) that infer capacityregularized network architectures from the data and require neither cross-validation nor fine-tuning when training the model. One of the two essential components of a PBDN is the development of a special infinite-wide single-hidden-layer neural network, whose number of active hidden units can be inferred from the data. The other one is the construction of a greedy layer-wise learning algorithm that uses a forward model selection criterion to determine when to stop adding another hidden layer. We develop both Gibbs sampling and stochastic gradient descent based maximum a posteriori inference for PBDNs, providing state-of-the-art classification accuracy and interpretable data subtypes near the decision boundaries, while maintaining low computational complexity for out-of-sample prediction.

Review:
The paper introduces a new type of (deep) neural network for binary classification. Each layer is in principle infinitely wide but in practice finite number of units is used. The layers are trained sequentially by first training one layer, and then always the next layer after the previous one. The main claim is that the proposed model gives comparable results to the alternative approaches by utilizing fewer hyperplanes that results in faster out-of-sample predictions.   The approach seems somewhat novel and the results support the claim to some extent. However, I found the paper difficult to read in many places (some specific comments below). Furthermore, the accuracy is essentially the same as for the other methods, the computational cost at the training time is not considered and other techniques used to reduce computations at prediction time are not discussed (see below). My overall feeling is that the paper has some potential, but the current exposition of the material requires too much effort from the reader, and hence the paper does not quite meet the standards of NIPS.    Some specific comments/questions:  - I felt the introduction of the iSHM is unnecessarily complicated. The model is actually far more simple than made to appear. It feels that Section 2 could have been both compressed and made clearer simultaneously, as now it contains many not-so-important mathematical details that simply confuse the reader rather than clarify things. The same applies also to Section 3 to some extent.  - All the equations have too little space which makes reading the text and formulae somewhat uneasy.  - I found Fig. 1 extremely hard to follow. Not only are the figures so small that they are barely readable, but the subfigure labeling is also confusing; the outer subfigure names are 1), 2), c) and d) (why not 1-4?), and these are also divided further into 'subsubfigures' named a)-f). It took far too much effort to try to understand the message of these figures. Furthermore, this figure is not even discussed in the main text: "We provide detailed explanations for Fig. 1 in Appendix D." Partly due to these reasons I never got a good intuition about how the model actually works and why it would be better than the traditional ones.  - I would expect the proposed model to be outperformed by convolutional networks (CNN) in image classification. This is not to say that it wouldn't be useful in some other settings, but since the authors claim (in the abstract) that the method achieves "state-of-the-art classification accuracy", I'm simply curious whether the authors agree about this intuition?  - The authors focus on the computational complexity for out-of-sample predictions. What is the computational cost at training time compared to the more traditional networks (e.g. those you used in the comparisons)?  - If the purpose is to reduce the computations at prediction time, there are also other methods specifically designed for this, namely those that attempt to replace the complex architecture by a simpler one, such that the simple model utilizes information from the complex one [e.g. 1, 2]. To my knowledge, these have been quite successful and would be interesting to know how your method compares to these approaches.  - Is it straightforward to extend the proposed model to multiclass problems or regression?   Typos / other minor points:  - 'nueral' page 2, line 63 - abbreviation iSHM introduced twice (page 2, first on line 57 and second time on line 81) - 'refereed' -> 'referred'? page 7, line 252   References:  [1] Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 535–541. ACM, 2006.  [2] Geoffrey Hinton, Oriol Vinyals, Jeff Dean. Distilling the Knowledge in a Neural Network. https://arxiv.org/abs/1503.02531   *** Update *** I thank the authors for submitting a rebuttal which addressed all my questions. After taking some more time to read the paper again I've changed my mind and now think that this could be worthwhile contribution to the conference. I have updated my overall score accordingly. I do, however, encourage the authors to take my initial comments - especially those regarding the readability - into account when preparing the final version of the paper.