Paper:
Posterior Concentration for Sparse Deep Learning
We introduce Spike-and-Slab Deep Learning (SS-DL), a fully Bayesian alternative to dropout for improving generalizability of deep ReLU networks. This new type of regularization enables provable recovery of smooth input-output maps with unknown levels of smoothness. Indeed, we show that the posterior distribution concentrates at the near minimax rate for α-Hölder smooth maps, performing as well as if we knew the smoothness level α ahead of time. Our result sheds light on architecture design for deep neural networks, namely the choice of depth, width and sparsity level. These network attributes typically depend on unknown smoothness in order to be optimal. We obviate this constraint with the fully Bayes construction. As an aside, we show that SS-DL does not overfit in the sense that the posterior concentrates on smaller networks with fewer (up to the optimal number of) nodes and links. Our results provide new theoretical justifications for deep ReLU networks from a Bayesian point of view.

Review:
This paper translates some theory of posterior concentration from Bayesian non-parametric regression to deep ReLU networks. This allows for the development of spike-and-slab priors on the weights of the hidden units that both provably concentrates around the true posterior distribution and ought to allow for automatic choice of hidden units. The main proof of posterior concentration follows related work in asymptotic statistics by considering the size of neighborhoods in \norm{.}_n, where n is the number of data points, and considering the limit of n-> \infty.  Originality: This seems to be the first paper to apply non-parametric regression techniques to prove posterior concentration properties for spike-and-slab priors on DNNs. This is relatively novel, even if the results are a natural extension of existing work on spike-and-slab variable selection.   Significance: The results seem to be a quite natural extension of existing non-parametric regression results; however, this could be a useful addition to the community as it informs potentially more interesting prior choices for researchers applying Bayesian methods to DNNs.   Clarity: Presentation wise, the paper seems relatively well structured; however, too much space is spent defining terms and introducing non-parametric regression concepts, resulting in too little space (1.5 pages) being devoted to the new results. Even less space is spent explaining the importance of these results. Additionally, discussion as to how these results can be applied for practitioners is quite lacking.   However, the exposition on posterior concentration and spike-and-slab priors is quite nice, well-written, and easy to follow. One suggestion is that the introduction section could be condensed quite a bit, with the relevant references being pushed into the next sections on deep learning and deep ReLU networks.   Quality: The proofs in the supplementary material do seem to follow logically, and the comments that I have are mostly minor. The work would be made much stronger by the inclusion of an experiments section showing how these DNNs could be trained.  Comments: - A tighter bound on function approximation for deep ReLU networks does exist than is given in Section 5.1, as [2] show that a network of depth k and width O(n/k) exists to approximate any function on n data points in p dimensions. Perhaps that argument could be adapted to include sparsity and thus show a tighter bound in Section 5.1, allowing fixed depth and layers, rather than the requirement that the number of layers grow as a function of data. - One primary weakness in this work is in the lack of experiments using the spike-and-slab theory. It would be very nice to see a) that the spike-and-slab priors discussed could be trained in practice, whether by some sort of Gibbs sampling, MCMC, or by a variant of standard SGD, and b) if an architecture could be adaptively trained using the prior on network width using Equation 18. - It is interesting that the composed deep network suggested by [1] (~180 parameters) performs so much better than the wide shallow network (~8000 parameters). Perhaps the authors can comment on if they would expect this result to hold in the limit of width as the single layer neural network converges to a Gaussian process? - The results seem to hinge on the true generative model, P_f0^n, being in the same class of functions as the model. Could the authors clarify if this is the case, and if not, what would happen to both sparsity and the posterior distribution? - Proof of Theorem 5.1: In Section 3, the authors point to the fact that they must verify all three conditions (Equations 11-13); however, in the proof, they only verify Equations 11 and 12. A quick sentence demonstrating why they do not need to verify Equation 13 would be quite helpful. - Corollary 5.1: In line 274, s_n = n \epsilon_n^2, which has been assumed to approach infinity in the limit (line 183). Thus, it seems like the corollary is just showing that the sparsity level is finite (and thus the number of non-zero parameters is finite), when it should be finite by construction of the neural network. Could the authors comment on this interpretation?  Updates after Author Responses: I thank the authors for addressing my mathematical comments, questions about implementations, and comments about paper structuring. These resolve many of my concerns with the paper.   Minor Comments/Typos: Line 69: Similarly as dropout, -> Similarly to dropout Line 79: Section 3 defined … -> Section 3 defines (rest of paragraph is in present tense) Equation 7: A verification that this holds for all L>1, p>1 and N>1 would be nice. It is possible to find nonsensical parameter settings that do not satisfy this inequality under those conditions, but I could not find a counter-example to the inequality. Line 168: Potentially (T \choose s)^{-1} instead of 1/ (T \choose s) for clarity. Also applies to other locations. Line 178: The posterior distribution should also include dependence on X (the observed features) and not just Y (the observed classes). Line 201: A citation for “it is commonly agreed …” would be quite helpful. Line 297: We proof… -> We prove… Line 297: There seems to be a missing \Epsilon. log N (…) -> log N \Epsilon (…) Line 307: Where does \epsilon_n/2 come from? It seems like the triangle inequality would give \epsilon_n instead.  References: [1] Mhaskar, H., Liao, Q., and Poggio, T. A. When and why are deep networks better than shallow ones? AAAI, 2017. [2] Zhang, C., et al. Understanding Deep Learning Requires Rethinking Generalization, ICLR, 2017. (see particularly Theorem 1 and Lemma 1 in Appendix C.)