Paper:
Deeply Learning the Messages in Message Passing Inference
Deep structured output learning shows great promise in tasks like semantic image segmentation. We proffer a new, efficient deep structured model learning scheme, in which we show how deep Convolutional Neural Networks (CNNs) can be used to directly estimate the messages in message passing inference for structured prediction with Conditional Random Fields (CRFs). With such CNN message estimators, we obviate the need to learn or evaluate potential functions for message calculation. This confers significant efficiency for learning, since otherwise when performing structured learning for a CRF with CNN potentials it is necessary to undertake expensive inference for every stochastic gradient iteration. The network output dimension of message estimators is the same as the number of classes, rather than exponentially growing in the order of the potentials. Hence it is more scalable for cases that involve a large number of classes. We apply our method to semantic image segmentation and achieve impressive performance, which demonstrates the effectiveness and usefulness of our CNN message learning method.

Review:
Given CRF potentials, one can write down the marginal predictions after only passing the initial messages from factors to variables. This paper incorporates that parametric form into a neural network, and fits it directly to minimize training error. This architecture appears to be empirically successful on a meaningful benchmark. I found the framing of the work partially misleading: As far as I can tell, the cost function reported doesn't care about structured prediction just pixel-wise errors, and no CRF model is actually fitted.To me "structured CRF prediction" means that there is a joint distribution over the labels given an input. Given such a model, it would be possible to fantasize joint plausible configurations of the labels, or report the jointly most probable labeling.I don't think the "Intersection over Union" score cares if the vector of labels is jointly plausible. The way to maximize this score is to predict the most probable label for each pixel independently. Learning a CRF model would learn a joint distribution, but here predictive performance, which doesn't care about the joint distribution, is maximized, not a CRF model. It isn't even clear to me if the "messages" in this paper will always correspond to realizable factors(?), let alone likely ones.The work is certainly interesting. The idea is a neat one, and seems to be an excellent heuristic for setting the parametric form of the output layer of a neural network to get good performance. I simply found some of the framing misleading and genuinely confusing on first reading.  line 332: "One advantage of message learning is that we are able to explicitly incorporate the expected number of inference iteration into the learning procedure." -- However, only one iteration is ever tried, and I presume implemented. I'm not convinced the details have been worked out or investigated enough to make this claim. Typo: line 54: massage -> messageThere's quite a few incorrect word endings, usually singular/plural disagreement. I lost motivation to note them down after noticing a spell checker hadn't been run: line 66: potenials line 120: sematic line 265: convlutional