Paper:
Bayesian Optimization for Probabilistic Programs
We present the first general purpose framework for marginal maximum a posteriori estimation of probabilistic program variables. By using a series of code transformations, the evidence of any probabilistic program, and therefore of any graphical model, can be optimized with respect to an arbitrary subset of its sampled variables. To carry out this optimization, we develop the first Bayesian optimization package to directly exploit the source code of its target, leading to innovations in problem-independent hyperpriors, unbounded optimization, and implicit constraint satisfaction; delivering significant performance improvements over prominent existing packages. We present applications of our method to a number of tasks including engineering design and parameter optimization.

Review:
The paper considers Bayesian optimization for probabilistic programs (PPs). The specific interest is in marginal maximum a posterior (MMAP) estimate for a subset of the parameters in the model defined by a PP. Large parts of the paper are devoted to describing the programmin workflow and the specific PP Anglican. 
The paper presents a implementation of Bayesian optimization to an interesting problem. I liked the content of the paper in general but not the presentation and had some doubts on the methods (as detailed below). However, the dislike of presentation might be my personal preference.   I'm not expert on Probabilistic Programming systems in broad sense (I'm confident in using STAN and few others). Hence, I find it very difficult to follow the presentation which relies on syntax for Anglican. I'm sorry but I don't fully understand the syntax in figures 2 and 3 and the long paragraphs of syntax style text - or at least I would need to first study the respective manual. For example in lines 119-120, I don't understand what "(sample:id dist)" means. However, the "intellectual" content in figures and text is syntax independent - and I think it should be in order to justify publication. For this reason I think authors should revise the paper and explain what they do in plain words and standard mathematics notations. For example, there is no need to use awkward "[:power-dis]" to denote distribution when authors could just write \theta \sim p(\theta).  I think the paper does not read very well in general. Authors use very condensed sentences that are hard to understand. For example, I did not understand the following: - line 144: "...sampling tested \theta values..." What are "tested \theta values"? - The list in lines 143-151 is too vague at this point. Could you, at least, refer to later sections when talking about things not defined yet (e.g. line 148). - line 200: "...combination...", what kind of combination; sum, product, ...? - line 201: "...significantly higher weigth". What is significant? How do you give weight to a kernel; do you have a prior over them and you sample GP's with different kernels? - line 204: What does mean "...linearity of GPs is exploited to convert the produced samples into a single GP."? - line 226: "If points are observed... target for the GP..." what are the points referred here and how are they observed? And what is target for the GP? - In line 166 and later authors talk about "problem independent hyperprior" but they never define that. The black box nature of their algorithm is an essential novelty here and the hyperprior plays major role there. Hence, it should be carefully justified.  Then a more fundamental question. In the Introduction and Discussion (and in few other places) authors talk about searching for marginal maximum a posterior estimate but as far as I've understood correctly they search for maximum marginal likelihood. In line 143 authors talk about "A code transformation allowing the marginal likelihood (ML) to be evaluated at a fixed \theta". This is also what seems to be possible with algorithm in Figure 3. I cannot find details on where the prior for \theta comes into play in this algorithm. This might be a problem of presentation that I criticized above and, if so, should be corrected. One specific point is line 208 where \hat{u}_m^* is defined to be the maximum expected posterior density of \theta. However, earlier in line 138 it was defined to be _log_ marginal _likelihood_.  I am a bit skeptical that the idea presented in section 4.4. works in general setting. How do you justify the choice of r_{\infty} for example? Let me tell an example from a real problem I'm working with currently. I'm interested in finding the (real) maximum a posterior estimate for \theta \in \Re^2. The prior is really vague and likelihood is really narrow leading to a situation where the meaningful posterior mass would cover only about 1% of authors hypercube. More spefically, the hypercube (in log \theta space) would be [0 1] x [0 1] and the MAP is around [0.9 0.9] and log p([0.9 0.9]+\epsilon*[1 1]|Y) + 10 < log p([0.9 0.9]|Y)  for all epsilon greater than 0.1. Hence, the posterior mass is practically negligible if the difference between log posteriors is greater than 10 and a naive BO algorithm failes here. Another question relates to the dimension of \theta. How large can it be?  Few minor specific comments: - Figure 3 caption: what does "probability p(y)=y according to value y = (C, \theta)" mean? Moreover, "C" is introduced much later than its first appearance in text or in reference to Figure 3.  - line 137: "\hat{u}_m^*" the same thing is denoted by \hat{w} in line 195 - line 138: "log marginal likelihood log p(\hat{\theta}_m^*|Y)"? isn't this log marginal posterior? - line 163: definition/explanation of q-cond comes much later so I was at first confused here. - the font size in figures 5 and 6 is too small  To summarize. I think the paper puts too much emphasis on "coding" before assuring that the theory behind the method really works. The paper could make a good example if formulated so that it concentrates on the method.