Paper:
Gradient-free Hamiltonian Monte Carlo with Efficient Kernel Exponential Families
We propose Kernel Hamiltonian Monte Carlo (KMC), a gradient-free adaptive MCMC algorithm based on Hamiltonian Monte Carlo (HMC). On target densities where classical HMC is not an option due to intractable gradients, KMC adaptively learns the target's gradient structure by fitting an exponential family model in a Reproducing Kernel Hilbert Space. Computational costs are reduced by two novel efficient approximations to this gradient. While being asymptotically exact, KMC mimics HMC in terms of sampling efficiency, and offers substantial mixing improvements over state-of-the-art gradient free samplers. We support our claims with experimental studies on both toy and real-world applications, including Approximate Bayesian Computation and exact-approximate MCMC.

Review:
Summary: The paper is primarily concerned with a Bayesian inference setting in which only unbiased estimates of an unnormalized density are available. The authors propose using kernel methods to approximate the unnormalized density as well as the the gradient of the log-probability in order to perform approximate HMC. The paper uses approximate versions of score matching to approximate the log-probability. Because standard score matching scales poorly with the number of samples being used (that is, the number of samples in the Markov chain thus far), the authors propose only using a random subset of the the samples (KMC lite) or a finite approximation of the infinite-dimensional RKHS (KMC finite). The paper has a number of intriguing ideas and potentially very useful methodologies for pseudo-marginal and ABC-type settings. However, there a substantial issues in the presentation of the methods and in the experimental validations. Major issues: 1. A crucial idea in the paper is to use (approximate) score matching to construct a nonparametric estimate to the (gradient of the) log-probability. However, the authors only describe score matching in the appendix, but at the same time repeatedly refer to the key score matching equation in that appendix. This is rather poor form, and room for the content of appendix A.1 needs to be found in the main paper. More generally, the authors have a bad habit of leaning heavily on the reader to understand a number of other recent papers, in particular their refs [12, 13, 14], and to a lesser extent [3]. I'm sympathetic to the fact that the NIPS paper format offers very limited space, but some of the relevant material (e.g., the details of the experimental setup of [12]), could easily to put in the supplementary material. Extra room could be found by, e.g., removing the proof sketch of Prop 3 and the useless paper outline at the end of the intro. 2. The authors state (line 94+): "we generally assume that [the density of interest] \pi is intractable, i.e. that we can neither evaluate \pi(x) nor \grad \pi(x) for any x, but can compute unbiased estimates of \pi(x)." However, when presenting their KMC method, eq. (3) makes use of the true Hamiltonian, i.e., it requires access to \pi(x). In the experiments section they state they are in fact using PM-MCMC in place of (3), which is fine given the working assumptions. However, this is never made clear earlier, and, moreover, the authors never bother to state the PM-MCMC acceptance criterion they are using. 3. The experiments were difficult to follow and, insofar as I could follow them, were unconvincing. a. Stability in high dims: it was unclear how many samples were used (i.e., the value of t), and how these samples were generated since apparently there was "no MCMC yet." Targeting a standard Gaussian here is extremely uninteresting and not even close to representative of what a real high-dimensional target might look like. Placing a distribution over the eigenvalues of the Gaussian covariance, then generating a number of Gaussians from that distribution would be much more reasonable. Also, it's odd to start from all the origin since in high dimensional centered Gaussians the origin is in a region of increasingly low probability. b. Mixing on a synthetic example: why only investigate mixing? I would expect burn-in to be more problematic since early samples used to estimate pi(x) are going to be unrepresentative. Thus, understanding how KMC performs during burn-in is crucial, and the authors themselves emphasize that the lite and finite versions will perform very differently while in the tails. Small point: min/median/max ESS would be more informative than mean ESS. c. GP classification: This task seems ok, though it would be good to include further experimental details (e.g., model used, source of benchmark samples) in the supplemental material. How much additional computation is required for the selection of kernel width via x-validation? Needing to run this x-validation seems potentially problematic. Why was 45% acceptance targeted? Why does the KMC MMD stop decreasing after ~3000 iterations in Fig 4(left)? Was KMC finite also tested for this problem? d. ABC: I found it extremely difficult to follow the set-up here. What is the "KMC-ABC" algorithm being tested? (As in 3c, Alg 1 as written is not applicable.) I eventually figured out that data was sampled from p(y | alpha = theta = 10 * \vec 1), then theta was inferred using the two ABC algorithms. But this was not clear on first reading and further details are missing. E.g., how many data points were used to form the posterior? Even more importantly, the skew-normal distribution used to test KMC-ABC is very simple and unrepresentative of the problems ABC is applied to.Minor points: 1. The fact that KMC lite falls back to a random walk seems like a really nice feature of that approach and the sort of important detail that might convince a practitioner to try out KMC ("when the method doesn't know what to do, it falls back to method I understand will do something sensible!"). Therefore, it would be good to add a sentence at line 301 explaining how/why this reversion to RW MH occurs. 2. It is of course well-known that choosing reasonable kernel hyperparameters is very important, since they typically encode scale/smoothness assumptions. Particularly in the very complicated models for which only unbiased estimate of pi are available or for which only forward simulations can be performed, the smoothness may vary substantially throughout the parameter space. The paper only addresses these sorts of issues in passing at the end of section 5. Other comments: Kamiltonian Monte Carlo is a terrible name. Why not just "kernel HMC"?l.126: missing comma in Hamiltonian l.224: with => are given byRESPONSE TO AUTHORS: I appreciate the authors' thoughtful response and efforts to redo the Gaussian experiments. However, there are substantial enough issues with the presentation of the methods and the experiments that I think the paper should be re-reviewed. I very much like the paper - the ideas in the paper are very interesting and they deserve to be presented in a clear manner so that the community can benefit from them as much as possible.