Paper:
Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning. A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agent's uncertainty about the environment. Computing a Bayes-optimal policy is however intractable for all but the smallest tasks. In this paper, we introduce variational Bayes-Adaptive Deep RL (variBAD), a way to meta-learn to perform approximate inference in an unknown environment, and incorporate task uncertainty directly during action selection. In a grid-world domain, we illustrate how variBAD performs structured online exploration as a function of task uncertainty. We also evaluate variBAD on MuJoCo domains widely used in meta-RL and show that it achieves higher return during training than existing methods.

Review:
*EDIT: Score increased after discussion with authors clarified many concerns raised below*
Summary:
This paper presents an algorithmic approach toward learning Bayes Optimal policies under the uncertainty of the environment. Leveraging meta-learning, the proposed variBAD approximate inference procedure is capable of adapting within the first episode at, what the authors term, meta-test time. 
Comments:
It is my estimation that this paper is well positioned to further current state-of-the-art adaptive RL frameworks or methodologies, whether they are meta-learned, transferred or directly inferred through probabilistic mechanisms. The primary contribution of this paper is in how variBAD learns the variational inference procedure. As noted in the related work section, many contemporary policy learning approaches via variational inference are limited by their construction, selection of prior distributions, etc. The advantage of the proposed methodology is that it is capable of efficiently inferring the current environment and adapting the policy learning procedure accordingly. The experiments successfully compare with relevant baselines and prior approaches. The discussion is well framed in highlighting the benefits and limitations of the proposed methodology in relief to prior approaches. One possible weakness in the experimentation, given how closely RL^2 matches the performance of variBAD, is that the specific contributions of each architectural choice or optimization protocol are unclear. There are a few areas in the paper where the authors suggest that ablating their model in specific ways would recover the core approaches present in RL^2. It would be instructive to see how/if performance degrades or converges toward that of RL^2 as the variBAD methodology is ablated.
The paper is well grounded in the literature, albeit skewed perhaps a bit too far toward recent meta-learning results. This is understandable given the focus of this paper, however there are other approaches that might deserve a mention as they similarly parameterize variation over possible MDPs with some latent variables. Namely, I have in mind a few lines of research such as Contextual MDPs (Hallak, et al, 2015; Jiang, et al, 2017; Dann, et al, 2018; etc.), Successor Features (Barretto, et al, 2016,2017,2019; Lehnert, et al, 2017; etc.)  and HiP-MDPs (Doshi-Velez and Konidaris, 2016; Killian, et al, 2017). In particular use of the HiP-MDP framework, Yao, et al (2018) also use the inferred latent variable used to identify the task to condition the policy. While it's always easy to dig into the rabbit holes of related research and distract from the overall objective of a paper, I thought that there was sufficient overlap with these other lines of research that the authors may find interesting. I do not claim that any one of these additional sources of prior work have been overlooked to the detriment of the current paper, they are offered as merely a suggestion to broaden the author's anchoring in the literature.
Now, some more specific questions about the paper and proposed approach. Further clarity along any of these questions would greatly improve the presentation of the paper as well as further convince me of the paper's suitability for publication.
1) What is the advantage of decoding the entire trajectory? It is well understood that this is advantageous in training as that data is available and allows for better inference of the variational parameters. However, under test conditions where the framework may be operating in environments that lie outside the distribution of MDPs it was trained on, I can imagine that errors in trajectory prediction may compound and throw off the entire inference procedure. The experimental set-up did not allay these concerns as there was no mention for holding out-of-distribution tasks/environments aside and the variation in environments is pretty narrow. 
2) How is the proposed trajectory decoding more stable than model predictive control? Is stability a large consideration for variBAD when exploring? How much can one trust the exploratory actions under variBAD? 
3) The visualization and careful explanation in Section 5.1 of how variBAD executes inference and learning was greatly appreciated. However, are these intuitions valid when extending beyond discrete state and action spaces? Can one make the same claims about the overall approach or procedure in the MuJoCo domains? It was mildly disappointing that a similar explanatory effort was not made in more complex environments. Even an acknowledgement of this being unreasonable would help round out the discussion in Section 5.2.
4) It is not clear what the connection is between the horizon H and the number of rollouts used for evaluation/inference/training. I spent a bit more time than necessary going over and over these items in the paper to where I think that I may understand but I'm still not 100% confident about what is impacted by the number of rollouts used.