Paper:
Meta learning has been making impressive progress for fast model adaptation. However, limited work has been done on learning fast uncertainty adaption for Bayesian modeling. In this paper, we propose to achieve the goal by placing meta learning on the space of probability measures, inducing the concept of meta sampling for fast uncertainty adaption. Specifically, we propose a Bayesian meta sampling framework consisting of two main components: a meta sampler and a sample adapter. The meta sampler is constructed by adopting a neural-inverse-autoregressive-flow (NIAF) structure, a variant of the recently proposed neural autoregressive flows, to efficiently generate meta samples to be adapted. The sample adapter then pushes meta samples to task-specific samples, based on a newly proposed and general Bayesian sampling technique, called optimal-transport Bayesian sampling. The combination of the two components allows a simple learning procedure for the meta sampler to be developed, which can be efficiently optimized via standard back-propagation. Extensive experimental results demonstrate the efficiency and effectiveness of the proposed framework, obtaining better sample quality and faster uncertainty adaption compared to related methods.

Review:
Summary of paper:
The authors propose a neural sampler for probabilistic models in the meta-learning setting. 
Their main claim is that their model captures uncertainty in samples better than competing methods and does so at lower cost.
In particular, they propose a scheme which separates sampling variables for a task tau into two components: a meta-sampler and a sample adapter.
The meta-sampler intuitively plays the role of  a learned conditional distribution over the target variables.
The sample adapter is a sampler which is seeded with samples from the meta-sampler and moves them towards the desired data-distribution based on a technique called optimal-transport Bayesian sampling.
Crucially, the meta-sampler is based on neural inverse autoregressive flows to have adequate representational capacity.
The authors use the proposed algorithm (denoted DAMS for distribution agnostic meta sampling) for a variety of tasks: 
First, parameters for logistic regression models are generated and evaluated on various UCI tasks.
Second, meta-learning over Gaussian mixture models (GMMs) is performed.
Third, posterior adaption in a toy regression task where the frequency parameter of a sine wave is estimated.
Last, the authors train neural networks with DAMS and test both classification accuracy as a function of test-time inference for test datasets of held out classes on cifar10, mnist and a few-shot training example on Mini-Imagenet, while also testing their method on meta-reinforcement learning Mujoco tasks.
Main Comments to authors:
Pros:
-interesting combination of techniques (IAF flows and WGF/Stein inference) to do meta-sampling
-empirically appealing results as pertaining to raw performance metrics like accuracy
Weaknesses:
- The evaluation is focused on #of steps during testing, but not on # of datapoints required for eval on new domains.
-> this is only half of what we care about when saying a sampler is "fast". The other half would be sample efficiency, which is typically the main motivation for Bayesian models and accurate uncertainty estimates, for instance when performing Bayesian Optimization. In fact, when making claims about uncertainty estimates as the goal of the paper, it would be most interesting to see how much test data the method needs to ingest before producing calibrated estimates. The method as currently presented only evaluates speed in terms of computation, but ignores sample efficiency entirely. As such, it is unclear from the given experiments to evaluate the main claim of the paper: that DAMS improves uncertainty adaptation.
-> the uncertainty is barely evaluated except in the low-d and toy sine wave illustration, which probably can be done equally well or better with regular posterior inference on D = D_train union D_test, i.e. using HMC. My suggestion to the authors would be to consider comparing to regular Bayesian inference (i.e. Monte Carlo/HMC/SVI) based on train_data and test data to compare to a ground truth estimate they might want.
-In Sec. 3.3 Eq. 6 and 7 a kernel is used and then not discussed much further. Kernels on high dimensional data (such as the weights of a NN)  are problematic to be used due to the curse of dimensionality. For the meta-sampler, even in the case of the multiplicative parametrization which lowers dimensionality, this would indicate that the kernel part of the objective might not be doing much work at all as in high enough dimensions all distances become even. If that were to be the case, the model might just look for the mode in any high-d example instead of actually sampling from a posterior and producing uncertainty estimates. Any empirical analysis and discussion on this is entirely missing here, unfortunately. As presented, the reader just has to accept that the objective functions make sense because the final product of putting all of the components together produces high accuracies. I would appreciate more details and careful analysis.
-Please also show the performance of meta-sampler with and without sample adapter in this case to clarify the effects of performing sample adaptation versus just using the meta-sampler, as this also is never compared in the paper. I.e. how good of a conditional model is the autoregressive flow? How much work does the sampler have to do? Would another conditional model do as well or worse? Why this choice of conditional model in particular if in the end sampling is put on top of it?
-The ELBO in Sec.3.4 involves an inference network over NN parameters (or potentially latent Zs per feature when using multiplicative parametrization). This object is highly nontrivial and not analyzed in terms of performance at all here.  Inference networks over neural networks are hard to get right and worthy of entire publications.
-Please clarify the prior used for BNN models.
-Please consider using HMC as a baseline for BNN models per task in terms of LLK to compare to DAMS.
Baselines:
- A lot of this paper relies on comparison to baselines, which are chosen to be mostly from the meta-learning field.
However, in practice the goal of the paper is Bayesian Inference in a particular class of models.
Hence:
- please consider adding conditional MNF as a baseline. This would clarify if a simple conditional version of MNF would suffice here compared to the involved scheme proposed in this paper and might show the advantage of DAMS over MNF (effectively the main driver for most of the experiments here).
-Similarly, please consider using conditional NAI flow as a baseline to see how far that gets the reader. 
Presentation Suggestions:
-You might want to consider establishing a formal relationship to a hierarchical probabilistic model with plates and show that this is just a way to perform sampling the posterior in a model like: P(y|x, tau) = integral_w P(y|x, w_t) P(w_t|tau) d_w
-This might help the flow of the paper by setting the stage early, as currently I had to read through it halfway to really understand the task before starting from the beginning to absorb the details of the proposed techniques.
-figures only readable in pdf, not in printout. Please enlarge fonts to give 'old school' readers a chance
-page 6  "..could be calculated effectively.." What does effectively mean here? Efficiently?
-page 6 under Theorem 1 typo: 'via the Eulaer scheme' -> Euler scheme
Related Work Suggestions:
- Consider citing "Predictive Uncertainty Quantification with Compound Density Networks" by Kristiadi et al, as it uses a conditional model with multiplicative parametrization successfully. I understand this is per data-point and the meta-learning scenario is focused on the per-dataset setting, but I find them related enough to consider a discussion.
- With regards to inference networks on BNNs, please cite "Latent Projection BNNs: Avoiding weight-space pathologies by learning latent representations of neural network weights" by Pradier et al, which attempts to do this and also discusses related work in more detail than this paper here. It is a hard task to be done well.
-The general form of the ELBO shown here is an instance of "Hierarchical Variational Models" by Ranganath et al, which should also be cited.
-Last but not least a recent paper in an ICML workshop on automatic machine learning  (https://sites.google.com/view/automl2019icml/accepted-papers) had a paper on "Improving Automated Variational Inference with Normalizing Flows" by Webb et al. This method looks a lot like a baseline method for this paper before task adaptation and WGF is considered and would effectively subsume the first batch of experiments entirely. I would propose the authors cite and discuss differences in detail.
Decision:
The paper uses a variety of 'puzzle pieces' that are quite involved on their own right. Putting them together and making it work is nontrivial and the authors demonstrate in their experiments that they get strong performance metrics. However, unfortunately, systematic ablation experiments and detailed analysis for the individual components used here and systematic comparisons to simple baselines are not performed. In addition, the paper is presented as a method for adaptive uncertainty quantification, which as argued above is not demonstrated empirically or else. What the paper does achieve is build a pipeline that gets high predictive performance on a meta-learning setting with lower computational requirements during testing than competing methods. I would suggest the authors focus on that aspect and add the required baselines that would clarify what ablations would do to the system and how the components interact.
As currently presented, I would argue for rejection since I am not sure of the scientific value of the interplay of components here as regarding uncertainty quantification. However, I think this paper is promising for a slightly different story with small experimental adjustments and would encourage the authors to consider that route.
(Edit Post Rebuttal:  revising score given the author response which addressed some of my concerns)