Paper:
Data Generation as Sequential Decision Making
We connect a broad class of generative models through their shared reliance on sequential decision making. Motivated by this view, we develop extensions to an existing model, and then explore the idea further in the context of data imputation -perhaps the simplest setting in which to investigate the relation between unconditional and conditional generative modelling. We formulate data imputation as an MDP and develop models capable of representing effective policies for it. We construct the models using neural networks and train them using a form of guided policy search [9]. Our models generate predictions through an iterative process of feedback and refinement. We show that this approach can learn effective policies for imputation problems of varying difficulty and across multiple datasets.

Review:
This paper connects methods for policy search in reinforcement learning with inference techniques for general generative models. Several inference methods are reviewed and extended. The second part of the paper applies these to imputation, which amounts to reconstructing damaged images from e.g. the MNIST dataset.This paper may seem very solid from a technical point of view, but at the same time it is not easy to read "story-wise". I can follow the technical steps in the paper (although I am not an expert in some aspects of the specific models), but for me many of these are described without enough "story". The introduction (and abstract) on the first page are very promising, but then the next five pages are very technical. This can be good, only if provided with enough intuition, such as is done in lines 207etc and 299-301, but there are too few of such fragments. Some of the insights (such as lines 125-126) are the key components of this paper, but they are motivated typically by looking at the formulas describing them. Problem statements such as equation 2 can be the right level of looking at similarities between problems (and solutions) but much of the remaining parts of (e.g.) sections in 2 are too detailed, IMHO. Then again some parts are very specific about upgrading specific techniques (such as 224-231) and performance comparisons, or about representations of policies (as two-layer networks with specific properties), although the paper (as described in the introduction) would be about sequential decision making for generative models (e.g. sampling).What I basically miss is a roadmap starting from the conceptual start of identifying where policy search and general inference meet, all the way through the various models and extensions. Note that I do appreciate the paper being very technical, and very specific/precise, but for me it does not seem always at the right level. The core aspect of the paper (data generation as sequential decision making) is not very novel, since this topic has been approached before (e.g. grammars and reinforcement learning based training of them, to name one example) but the solid way this paper approaches it (in the current state-of-the-art inference techniques) seems a good way to go. The second part of the paper (section 3) does a better job by first describing an intuitive model of imputation (the problem) and then providing technical details of the solution. (I would like to see some more intuition and description of the reward component though, since that may not be obvious to all readers, including me). I might even have the feeling that imputation would be a good problem to start the paper with, as an illustration of the proposed framework. Sections 3.1. and 3.2. are then somewhat unbalanced again in which details they describe and which should be "known" to the reader from existing literature. I'd rather read more intuition again (such as 299-302), and slightly less details. The details concerning the read and write operations in section 3.2. are not well enough described to understand/appreciate what happens here.Overall, the methods and the imputation setting are relatively clear when arriving in section 4. However, the introduction mentions that there are no alternatives to compare to, such that the authors compare internally, and against a baseline. I do not fully understand why one cannot find alternative techniques in the area of computer vision (for example Francis Bach's work, but there are others) when it comes to image reconstruction (in probabilistic/generative settings). The results seem "ok", but there's not much help to evaluate the findings. The authors state that "appear quite promising", and visually inspecting the results may verify that, but again this evaluation is not totally conclusive. At the end of section 4 it seems that performance on this task is important, whereas before it seemed that comparing the extension (of the LSTM model) against the original was important, whereas even earlier it seemed that connecting generative models and sequential decision making was important. Maybe these different goals, with different levels of description, are the weaknesses of the paper. Overall, a solid technical approach with a description that is maybe not at the right level (of detail). The writing is clear, polished, and very precise. In addition, the experimental results are nice, and they show the viability of the approach, but they may not necessarily demonstrate how to pick which method here, and they do not necessarily exemplify the main thesis of the paper (see the title). I guess that it would be much better to transform parts of this paper to focus more on ideas and to provide a heavy technical report with all the details concerning specific Bernoulli distributions and all that. Nevertheless, even in this form it is an interesting paper.