Paper:
Structure from motion (SfM) is an essential computer vision problem which has not been well handled by deep learning. One of the promising trends is to apply explicit structural constraint, e.g. 3D cost volume, into the network. In this work, we design a physical driven architecture, namely DeepSFM, inspired by traditional Bundle Adjustment (BA), which consists of two cost volume based architectures for depth and pose estimation respectively, iteratively running to improve both. In each cost volume, we encode not only photo-metric consistency across multiple input images, but also geometric consistency to ensure that depths from multiple views agree with each other. The explicit constraints on both depth (structure) and pose (motion), when combined with the learning components, bring the merit from both traditional BA and emerging deep learning technology. Extensive experiments on various datasets show that our model achieves the state-of-the-art performance on both depth and pose estimation with superior robustness against less number of inputs and the noise in initialization.

Review:
The paper tackles Structure from Motion, one of the canonical problems in computer vision, and proposes an approach that brings together geometry and physics on one hand and deep networks on the other hand. Camera unprojection and warping (of depth maps and features) are used to build a cost volume onto hypothetical planes perpendicular to the camera axis. Similarly, various camera poses are sampled around an initial guess. A deep network regresses form the cost volume to a camera pose and a depth map. The method can be applied iteratively, using the outputs of the current stage as the initial guess of the next one. Training is supervised, and the the results are evaluated on multiple datasets.
I am inclined to recommend accepting the paper for publication, because it addresses a canonical problem, outperforms the state of the art on multiple datasets and brings together geometry / physics and deep learning, which is IMO very a promising and underexplored direction.
I found the method section a bit difficult to read though, and even after several readings I cannot get my head around it. Specifically, here are some issues that I hope the Authors could clarify.
1. In Sec. 3 the Authors write "We then sample the solution space for depth and pose respectively around their initialization". However in Sec 3.2 they write "we uniformly sample a set of L virtual planes {dl} Ll=1 in the inverse-depth space". In what way are the planes "around their initialization"? If the initial depth map spans over multiple orders of magnitude, will the planes be uniformly sampled between the minimum and maximum disparity of the initial map? If yes, it seems that the initial depth map is not really needed, just its minimum and maximum value is needed, but then how come the method can be applied iteratively with respect to depth?
2. The Authors mention that depth maps are warped onto the virtual planes using differentiable bilinear interpolation. Is there a mechanism to protect from interpolating across discontinuities? If no, were bleeding edge artifacts observed?
3. In the introduction, the Authors point that prior methods have trouble dealing with textureless, reflective or transparent approaches, but it's not clear form the paper where it addresses these cases, and if yes, what is the mechanism for that.
Lastly, if the authors are not planning to release the code, the implementation details section is a bit too high-level and does not contain enough details to reimplement the Author's technique. For example, "our network learns a cost volume of size L × W × H using several 3D convolutional layers with kernel size 3 × 3 × 3"  - more details about this network are needed, as well as the others in the paper.