Paper:
We propose Bayesian hypernetworks: a framework for approximate Bayesian inference in neural networks. A Bayesian hypernetwork h is a neural network which learns to transform a simple noise distribution, p( ) = N (0, I), to a distribution q(θ) := q(h( )) over the parameters θ of another neural network (the "primary network"). We train q with variational inference, using an invertible h to enable efficient estimation of the variational lower bound on the posterior p(θ|D) via sampling. In contrast to most methods for Bayesian deep learning, Bayesian hypernets can represent a complex multimodal approximate posterior with correlations between parameters, while enabling cheap iid sampling of q(θ). In practice, Bayesian hypernets provide a better defense against adversarial examples than dropout, and also exhibit competitive performance on a suite of tasks which evaluate model uncertainty, including regularization, active learning, and anomaly detection.

Review:
This paper presents Bayesian Hypernetworks; variational Bayesian neural networks where the variational posterior over the weights is governed by a hyper network that implements a normalizing flow (NF) such as RealNVP and IAF. As directly outputting the weight matrix with a hyper network is computationally expensive the authors instead propose to utilize weight normalisation on the weights and then use the hyper network to output scalar scaling variables for each hidden unit, similarly to what was done at [1]. The main difference with this prior work is that [1] consider these NF scaling variables as auxiliary random variables to a mean field Gaussian distribution over the weights whereas this paper attempts to posit a distribution directly on the weights via the NF. This avoids the nested variational approximation and auxiliary models of [1], which can potentially yield a tighter bound. The proposed method is evaluated on extensive experiments.
This paper seems like a plausible idea with extensive experiments but the similarity with [1] make it an incremental contribution and, furthermore, it seems that it has a technical issue with what is explained at Section 3.3. More specifically, if you generate the parameters \theta according to Eq. 7 and posit a prior over \theta then you will have a problematic variational bound as there will be a KL divergence, KL(q(\theta) || p(\theta)), with distributions of different support (since q(\theta) is defined only along the directions spanned by u), which is infinite. For the KL to be valid you will need to posit a prior distribution over `g`, p(g), and then consider KL(q(g) || p(g)), with q(g) being given by the NF. From the experiment paragraph at page 5 though I deduct that you instead employ “an isotropic standard normal prior over the weights”, i.e. \theta, thus I believe that you indeed have a problematic bound. How do you actually compute logq(\theta) when you employ the parametrisation discussed at 3.3? Did you use that parametrisation in every experiment?
Other than that, I believe that it would be interesting to experiment with a `full` hyper network, i.e. generating directly the entire parameter vector \theta, e.g. at the toy regression experiment where the dimensionality is small. This would then better illustrate the tradeoffs you make when you reduce the flexibility of the hyper-network to just outputting the row scaling variables and the effect this has at the posterior approximation.
 
Typos:
(1) Page 3, 3.1.1 log(\theta) -> logp(\theta).
(2) Eq. 6, it needs to be |det \frac{\partial h(\epsilon)}{\partial \epsilon}|^{-1} or |det \frac{\partial h^{-1}(\theta)}{\partial \theta}| for a valid change of variables formula.
[1] Louizos & Welling, Multiplicative Normalizing Flows for Variational Bayesian Neural Networks.