Paper:
Variational Bayesian Monte Carlo
Many probabilistic models of interest in scientific computing and machine learning have expensive, black-box likelihoods that prevent the application of standard techniques for Bayesian inference, such as MCMC, which would require access to the gradient or a large number of likelihood evaluations. We introduce here a novel sample-efficient inference framework, Variational Bayesian Monte Carlo (VBMC). VBMC combines variational inference with Gaussian-process based, active-sampling Bayesian quadrature, using the latter to efficiently approximate the intractable integral in the variational objective. Our method produces both a nonparametric approximation of the posterior distribution and an approximate lower bound of the model evidence, useful for model selection. We demonstrate VBMC both on several synthetic likelihoods and on a neuronal model with data from real neurons. Across all tested problems and dimensions (up to D = 10), VBMC performs consistently well in reconstructing the posterior and the model evidence with a limited budget of likelihood evaluations, unlike other methods that work only in very low dimensions. Our framework shows great promise as a novel tool for posterior and model inference with expensive, black-box likelihoods.

Review:
Summary:  The paper considers variational inference in the case where likelihood functions themselves are expensive to evaluate. It suggests approximating the ELBO using probabilistic numerics. A Gaussian process prior is placed on the log joint of the model. A novel acquisition function is proposed along with an approximation of the ELBO for a variational mixture distribution based on the GP posterior and simple Monte Carlo for the mixture entropy. Empirical comparison is performed against a variety of relevant baselines. In meaningful synthetic experiments the proposed method outperforms other methods in higher dimensional cases. In a real example of a neuronal model the proposed models and its variants are the only ones that work.  Technical soundness:  Overall the paper is technically sound. I would have liked one more compelling real world example. I do also have some questions for the authors which would likely effect my score.  Acquisition function: I have some anxiety around the fact that the best performing acquistion function does not depend on the variational parameters (line 159) and that a method that ignores the variational aspect but keeps this search method can often give a better posterior approximation. This seems to run counter to the intuition in line 147 about a sequence of integrals under the corresponding variational approximations. Clearly there are useful things in the algorithm as a whole and the variational marginal likelihood approximation still dominates but should this have changed the emphasis of the paper?  Perhaps the (non-variational) search method in equation 8) is the real star? This also relates to some issues around the way the paper is written in the next section.  Variational mixture distributions: I have also experienced the difficulties the authors describe (footnote 2) with the mixture bound of Gershman et al 2012. Indeed my general impression was that mixture approximations are hard to get working even with MC approximations. Perhaps this is due to the fact (see Jaakkola and Jordan 1998 which the authors shouldd cite) that the improvement in ELBO is at best logarithmic in the number of mixture components and can therefore be lost beneath the Monte Carlo noise. Did the authors find that adding mixture components beyond two or three gave much improvement? Did they rule out that this was an optimization effect? Were the results sensitive to initialization?  Annealed importance sampling: I am surprised by how poorly AIS performed. My understanding from looking at the Gunter et al paper is that the proposal distribution used was a Metropolis-Hastings one. Did the authors look at multivariate slice sampling? This would likely be less sensitive to step sizes and indeed the authors proposed method calls slice sampling as a subroutine.  Clarity:  Apart from the questions discussed above the paper is well written. It references key literature. It is measured and resists hyperbole, carefully drawing the readers attention to potential limitations.  I do think that an expert would have a good chance of reproducing the key results given what is written.  I would have liked more discussion of equation 8. This seems to be a key point in the paper and has not received sufficient emphasis.  I would have really liked a visualisation of the algorithm in action in a case that was simple enough to plot. This would have helped answer my questions about the variatonal approximation above.  It is great that the authors are willing and able to release code after the review process. Why not release it in an anonymized repository at this stage next time? This would help build reviewer confidence in the submission.  I suggest that the authors include larger versions of Figures 1 and 2 in the supplement.  The references need tidying up. For instance in many cases the publication venue has been ommitted.  Originality and significance:  There is an interesting new idea here which I think could be important going forward. Specifically, I like the idea that probabilistic numerics and variational inference can be usefully combined. Given my concerns around the non-variational acquisition function I'm not sure it has been fully exploited yet. The solid experimental contribution should also be given credit.  References:  Gunter, Tom and Osborne, Michael A and Garnett, Roman and Hennig, Philipp and Roberts, Stephen J (2014), Sampling for Inference in Probabilistic Models with Fast Bayesian Quadrature, Advances in Neural Information Processing Systems 27 Gershman, S., Hoffman, M., & Blei, D. (2012) Nonparametric variational inference. International Conference on Machine Learning. Tommi S. Jaakkola , Michael I. Jordan  (1998) Improving the Mean Field Approximation via the Use of Mixture Distributions. Learning in Graphical Models.  Post author feedback comments:  I have read the author response. I am satisfied with their reply and have increased my score.