Paper:
Just-In-Time Learning for Fast and Flexible Inference
Much of research in machine learning has centered around the search for inference algorithms that are both general-purpose and efficient. The problem is extremely challenging and general inference remains computationally expensive. We seek to address this problem by observing that in most specific applications of a model, we typically only need to perform a small subset of all possible inference computations. Motivated by this, we introduce just-in-time learning, a framework for fast and flexible inference that learns to speed up inference at run-time. Through a series of experiments, we show how this framework can allow us to combine the flexibility of sampling with the efficiency of deterministic message-passing.

Review:
This paper is an extension of the recent Heess et al "Learning to Pass Expectation Propagation Messages". AIUI, Heess et al uses neural networks to learn an approximate mapping between input and output messages. In Hees et al, learning is done generically for each model, before specific problem instances are seen. Their approach has the drawback of requiring the compilation of synthetic training data, and being unable to adapt to model peculiarities during the message-passing phase. The present paper fixes both of these problems by learning input-output mappings as messages are being passed. It does this by falling back to an "oracle" for each message until the approximate mapping at that node has been suitably trained. The oracle is simply the (presumably expensive) way in which messages would have been computed in the absence of an approximation.The approach is tested on two synthetic problems, as well as on a probabilistic model of corn yield. Infer.NET is used throughout.The paper presents some important concepts, and I think it should be accepted because it is a "good enough" presentation of concepts which will have to be published at some point anyway. My main criticism is that the experiments seem biased in favor of the proposed method. For instance, the last subsection of Experiments says "We find that with around 100,000 samples the message for the yield factor can be computed accurately, making these by far the slowest computations in the inference procedure". The factor in Figure 5a looks pretty simple, I don't see why some sort of smooth approximation couldn't be used to compute messages going through it - 100,000 samples sounds excessive. At the same time, the proposed method for learning approximate message-operators based on random decision forests seems very slow. Perhaps another reviewer who is more familiar with expectation propagation and Infer.NET could comment on this.Perhaps also because I don't have a good background on expectation propagation, I find that after reading the paper I don't have a satisfactory intuition about why the proposed method, as well as the one in Heess et al, is needed. Do messages really jump around so much as they converge, that we need neural networks or random decision forests to speed this up? It would be nice if there was room to go into more detail about what's happening under the hood. Or am I missing something.Minor comments: "Regressor" should be defined, in Wikipedia it is defined as "independent variable", but I think the meaning is different in this paper. Expectation propagation (and Infer.NET?) should be mentioned in the abstract! "Related work" is usually Section 2, I am not sure why this paper puts it at Section 5.Clarity: The paper is fairly clear. Important details of experiments and models have been moved to Supplementary Material, which makes it a bit harder to understand what is happening in detail.Originality: The paper is original. It is to a certain degree an "engineering" paper, in that a number of different approximate techniques (EP, importance sampling, random forests) are drawn together without full justification. But I think this comes with the territory.I think I addressed quality and significance above.