Paper:
THE DEEP WEIGHT PRIOR
Bayesian inference is known to provide a general framework for incorporating prior knowledge or specific properties into machine learning models via carefully choosing a prior distribution. In this work, we propose a new type of prior distributions for convolutional neural networks, deep weight prior (dwp), that exploit generative models to encourage a specific structure of trained convolutional filters e.g., spatial correlations. We define dwp in a form of an implicit distribution and propose a method for variational inference with such type of implicit priors. In experiments, we show that dwp improves the performance of Bayesian neural networks when training data are limited, and initialization of weights with samples from dwp accelerates training of conventional convolutional neural networks.

Review:
Summary:
This paper proposes the ‘deep weight prior’: the idea is to elicit a prior on an auxiliary dataset and then use that prior over the CNN filters to jump start inference for a data set of interest.  Both explicit and implicit priors are considered, with the latter having the benefit of increased flexibility but having the drawback of a lack of a parametric form to plug in to the ELBO.  The authors address this last point by extending the ELBO appropriately.  Experiments are performed testing the prior’s ability to capture trained filters (Figure 1), provide a good initialization (Figure 2), improve sample efficiency (Figure 3), improve training speed (Figure 4).  
Pros:
I like this paper: it is a intuitive idea, and the experiments explore exactly what one would hope to gain from the prior (i.e. better initialization, improved sample efficiency).  I find the paper clearly written and to have a logical flow.  Furthermore, I think eliciting priors---while so crucial in more traditional Bayesian modeling---has been mostly overlooked by the Bayesian ML community, and this paper clearly shows that there are gains to be had from a fairly straightforward procedure.  
Cons:
The only potential issue with the paper is the use of the implicit prior, as it complicates variational inference, requiring the extension to the ELBO described in Section 3.2.  As far as I can tell, all experiments use the implicit priors.  I would have liked to have seen an experiment using a parametric prior (eg Gaussian) that shows what gains the implicit prior provides.  Or is it simply a matter of memory efficiency?  
Other comments:
-- Nice first sentence in the introduction!  I like how it’s a general statement but immediately focuses the reader’s attention to the paper’s topic.
-- While it doesn’t say so explicitly, the paper seems to imply it is the first to use implicit priors.  Some previous work that uses some form of implicit prior includes:
Runs a chain to refine the prior: Alex Lamb et al. "GibbsNet: Iterative Adversarial Inference for Deep Graphical Models." Advances in Neural Information Processing Systems. 2017.
Optimizes a NN implicit prior based on an invariance objective: Eric Nalisnick and Padhraic Smyth. "Learning priors for invariance." International Conference on Artificial Intelligence and Statistics. 2018.
Defines implicit priors over functions through samplers: Chao Ma, Yingzhen Li, and José Miguel Hernández-Lobato. "Variational Implicit Processes." arXiv preprint arXiv:1806.02390 (2018).
Evaluation:  I recommend this paper for acceptance.  It is a sensible idea with pointed experimental validation.