Paper:
Augur: Data-Parallel Probabilistic Modeling
Implementing inference procedures for each new probabilistic model is timeconsuming and error-prone. Probabilistic programming addresses this problem by allowing a user to specify the model and then automatically generating the inference procedure. To make this practical it is important to generate high performance inference code. In turn, on modern architectures, high performance requires parallel execution. In this paper we present Augur, a probabilistic modeling language and compiler for Bayesian networks designed to make effective use of data-parallel architectures such as GPUs. We show that the compiler can generate data-parallel inference code scalable to thousands of GPU cores by making use of the conditional independence relationships in the Bayesian network.

Review:
This paper introduces a new probabilistic programming language for Bayesian networks, Augur, and a compiler that are designed to produce parallel inference with GPU. The whole system is composed of two parts, the front end, which converts the model description embedded in Scala into a symbolic representation, and the back end, which generates CUDA code from the intermediate representation at runtime. Compared to other probabilistic languages, Augur is different in that it is designed to produce parallel inference method for scalability, and that it uses a compact symbolic representation for distributions.It is a meaningful contribution to automatically produce a parallel inference program for a general Bayesian model. Also, the LDA experiments addresses a common scalability problem in probabilistic programming that much more memory would be used in a probabilistic program than a hand-written program. I find it not difficult to understand the main idea of this paper, although a lot of details such as the construction of IR and the inference configuration in Augur are omitted.My main concern is about the efficiency and the overhead of the automatically generated CUDA code:First, it is not trivial to write a CUDA program to achieve high parallelization given a problem, and the efficiency depends highly on the number of independent tasks and the memory usage. The current proposed method seems to simply take a product expression in the symbolic representation and parallelize the computation on GPU. Is there a general mechanism for the compiler to detect which product should be parallelized and which should be rather run sequentially on CPU? Second, the authors discusses two algorithms to optimize the sampling of multiple Dirichlet distributions, and the compiler will take the better one according to runtime values.  Does this strategy require extra work to apply to a new distribution, or can you make it a general rule for all similar models?Lastly, AFAIU the CUDA code is generated once at the beginning of a MAP/Sampling call with a constant time cost, and the currently values of the model variables is used to optimize the generated code. However, values such as the number of subtasks for a product operation can depend on other variables and is changed during the execution. Would that mean a recompilation of the CUDA code? Another concern is the limited expressive capability of Augur. The currently language does not allow a variable size for the model support. So it can not be used for many model selection problems, e.g., a Gaussian mixture model with the number of clusters being a random variable with a prior distribution. Can author show that Augur supports all the Bayesian networks?The experiment results for the multivariate linear regression is not impressive. The parallelized code by Augur performs similarly to the sequential code by Jags. Also, the labels and plots in Figure 2(a) are unreadable. For the LDA experiments, it is well known that the collapsed sampler converges faster than the uncollapsed sampler as a function of the sample size. However, I am suprised that Figure 2(b) shows that the Factorie converges slower and also to a significantly lower value of the predictive probability than the other methods.--------------Updates after feedback:The authors feedback addresses my questions on the efficiency of the CUDA code to my satisfaction. However, their response on Factorie confirms my concern on the comparison between CUDA parallelized sampler and the collapsed Gibbs sampler. The plot of Factorie does not show the actual performance of collapsed Gibbs on the predictive probability. It is therefore hard to judge if Augur achieves the same performance as collapsed Gibbs given the same amount of time. I suggest the authors discusses this issue explicitly in a final version of the paper.