Paper:
Depth from a Single Image by Harmonizing Overcomplete Local Network Predictions
A single color image can contain many cues informative towards different aspects of local geometric structure. We approach the problem of monocular depth estimation by using a neural network to produce a mid-level representation that summarizes these cues. This network is trained to characterize local scene geometry by predicting, at every image location, depth derivatives of different orders, orientations and scales. However, instead of a single estimate for each derivative, the network outputs probability distributions that allow it to express confidence about some coefficients, and ambiguity about others. Scene depth is then estimated by harmonizing this overcomplete set of network predictions, using a globalization procedure that finds a single consistent depth map that best matches all the local derivative distributions. We demonstrate the efficacy of this approach through evaluation on the NYU v2 depth data set.

Review:
This paper proposed a novel monocular depth estimation algorithm. Instead of directly fitting the depth map, the method predicts a series of filter-bank response coefficients of the depth map. In the prediction stage the proposed method predict the filter response distribution and recover the depth through a global energy minimization step. 
In general the proposed method is technically sound. The general idea also makes sense in better capturing high-level statistics. The method seems novel. However, I have some concerns on how to prove it matters and why the proposed method is chosen.    The general idea is somewhat similar with some recent work which incorporates loss function over the high-order statistics on many low-level vision tasks such as flow, monocular depth recovery and image generation (e.g. "DisparityNet" Mayer et al. 2016, "Image Generation with Perceptual Loss" Ridgeway et a. 2015, "DeePSiM loss" Dosovitskiy et al. 2015, "Depth Estimation" Eigen et al. 2015). All the methods use loss functions that capture high-level statistics, or so-called perceptual loss through handcrafted (or learnable) filter map responses.   Motivated by this I think a baseline is required to convince the readers: instead of fitting filter-bank response then followed by an energy minimization stage to recover the depth, whether we can fit the depth using the following loss function: the coefficient similarity between the filter responses from GT depth and predicted depth. This scheme does not require the time-consuming alternating inference in prediction, but in the meantime captures the high-order statistics through the handcrafted filters. I wonder what is the performance and why the proposed is favored by the author.  The author claims the importance of fitting the coefficient of filter-bank responses, but did not to show such importance through experiments. Is this visually more appealing or better preserve 3D geometry than directly fitting depth map? Or does modeling the uncertainty brings diverse prediction? From section 4.2 I can hardly understand why it matters. From the comparison study, it seems even much less important than pretraining a VGG model on ImageNet or incorporating semantic labeling loss.   Moreover, there lacks of some foundation why fit high-order depth map statistics directly, rather than the 3D geometry space, and why such a series of handcrafted gaussian filters are used. In fact, it is totally doable that the author could incorporate the inference stage of eq. 6 into the network and conducted learning the filters in an end-to-end manner. In this way the objective of reconstructing depth could also directly utilized. In fact, I also have a doubt that the indirect regression might hurts the performance in terms of RMSE since the model is not optimized to minimize such a loss.   Overall I think the current stage of the paper is on the borderline and I am expecting the author's rebuttal to address my concerns.