Paper:
Bayesian Dark Knowledge
We consider the problem of Bayesian parameter estimation for deep neural networks, which is important in problem settings where we may have little data, and/ or where we need accurate posterior predictive densities p(y|x, D), e.g., for applications involving bandits or active learning. One simple approach to this is to use online Monte Carlo methods, such as SGLD (stochastic gradient Langevin dynamics). Unfortunately, such a method needs to store many copies of the parameters (which wastes memory), and needs to make predictions using many versions of the model (which wastes time). We describe a method for "distilling" a Monte Carlo approximation to the posterior predictive density into a more compact form, namely a single deep neural network. We compare to two very recent approaches to Bayesian neural networks, namely an approach based on expectation propagation [HLA15] and an approach based on variational Bayes [BCKW15]. Our method performs better than both of these, is much simpler to implement, and uses less computation at test time.

Review:
Paper Title: Bayesian Dark KnowledgePaper Summary: This paper presents a method for approximately learning a Bayesian neural network model while avoiding major storage costs accumulated during training and computational costs during prediction. Typically, in Bayesian models, samples are generated, and a sample approximation to the posterior predictive distribution is formed. However, this requires storing many copies of the parameters of a model, which may require a great deal of storage/memory in high-parameter neural network models. Additionally, prediction on test data requires evaluation of all sampled models, which may be computationally costly. This paper aims to learn a model that makes (approximate) posterior-predictive-based predictions on test data without storing many copies of the parameter set. The method presented here accomplishes this by training a "student" model to approximate a Bayesian "teacher's" predictions---a procedure in the neural networks literature referred to as "distillation" (and also referred to in previous papers as "model compression"). However, the student and teacher are trained simultaneously in an online manner (so no large collection of samples are required to be stored, even during training). Experiments are shown on synthetic data (a toy 2D binary classification problem and a toy 1D regression problem), on classification for the MNIST dataset, and on regression for a housing dataset. Comments:- I feel that working to develop computational tools for practical Bayesian inference in neural networks is an important direction of research (in particular, tools that mitigate the issue of large storage/memory requirements when sampling in large-parameters Bayesian models), and I feel that this paper is making good steps in this direction.- One issue I have with the novelty of this paper is that the fundamental concept being developed here (student/teacher learning, or model distillation/compression) is not new --- the authors apply this existing idea in a new domain. However, I do feel the main novelty (and strength) of this paper is the clever way in which the algorithm carries out simultaneous online training of the teacher and student without requiring storage of samples. This is an online distillation/compression method in which the teacher is never "fully trained" (the teacher never actually provides full posterior-predictive "labels" for the student) and yet the student is still able to learn the "full trained" teacher's model.- One potential computational issue with the presented algorithm is that the student must be trained (to mimic the teacher), on the training dataset D', at every iteration during the MCMC algorithm. I believe this requires a great deal more training of the student than in typical distillation methods (which do not need to make many passes through the training dataset D'). This might be particularly problematic if D' is required to be large in order to achieve good performance. In general, I feel that there is not enough discussion about the training set D' (whether particular choices about D' affect the method's performance, and the particular details used in the experiments presented in this paper).- I feel that the experiments in this paper were not totally polished and thorough. The authors write that they could not enable a proper comparison of all methods on all datasets because (in part) "the open source code for the EP approach only supports regression" and "we did not get access to the code for the VB approach in time for us to compare to it". I appreciate the honesty here! However, it would be nice to complete these goals and polish the experiments section in this paper.