Paper:
BA-NET: DENSE BUNDLE ADJUSTMENT NETWORKS
This paper introduces a network architecture to solve the structure-from-motion (SfM) problem via feature-metric bundle adjustment (BA), which explicitly enforces multi-view geometry constraints in the form of feature-metric error. The whole pipeline is differentiable, so that the network can learn suitable features that make the BA problem more tractable. Furthermore, this work introduces a novel depth parameterization to recover dense per-pixel depth. The network first generates several basis depth maps according to the input image, and optimizes the final depth as a linear combination of these basis depth maps via feature-metric BA. The basis depth maps generator is also learned via end-to-end training. The whole system nicely combines domain knowledge (i.e. hard-coded multi-view geometry constraints) and deep learning (i.e. feature learning and basis depth maps learning) to address the challenging dense SfM problem. Experiments on large scale real data prove the success of the proposed method.

Review:
I believe that the authors have a solid contribution that can be interesting for the ICLR community.
Therefore, I recommend to accept the paper but after revision because the presentation and explanation of the ideas contain multiple typos and lacking some details (see bellow). 
Summary:
The authors propose a new method called BA-Net to solve the SfM problem by explicitly incorporating geometry priors into a machine learning task. The authors focus on the Bundle Adjustment process. 
Given several successive frames of a video sequence (2 frames but can be extended up to 5), BA-Net jointly estimates the depth of the first frame and the relative camera motion (between the first frame and the next one).
The method is based on a convolutional neural network which extracts the features of the different pyramid levels of the two images and in parallel computes the depth map of the first frame. The proposed network is based on the DRN-54 (Yu et al., 2017) as a feature extractor. 
This is complemented by the linear combination of depth bases obtained from the first image.
The features and the initial depth then passed to the optimization layer called BA-layer where the feature re-projection error is minimized by the modified LM algorithm. 
The authors adapt the standard multi-view geometry constraints by a new concept of feature re-projection error in the BA framework (BA-layer) which they made differentiable. 
Differentiable optimization of camera motion and image depth via LM algorithm is now possible and can be used in various other DL architectures (ex. MVS-Net can probably benefit from BA-layer).
The authors also propose a novel depth parametrization in the form of linear combination of depth bases which reduces the number of parameters for the learning task, 
enables integration into the same backbone net as used or feature pyramids and makes it possible to jointly train the depth generator and the BA-layer. 
Originally the proposed approach depicts the network operating in the two-view settings. The extensibility to more views is also possible and, as shown by authors, proved to improve performance. It is, however, limited by the GPU capacity. 
Overall, the authors came up with an interesting approach to the standard BA problem. They have managed to inject the multi-view geometry priors and BA into the DL architecture. 
Major comments regarding the paper:
It would be interesting to know the evaluation times for the BA-net and more importantly to have some implementation details to ensure reproducibility.
Minor comments regarding the paper:
-	The spacing between sections is not consistent. 
-	Figures 1 is way too abstract given the complicated set-up of the proposed architecture. It would be nice to see more details on the subnet for depth estimator and output of the net. 
Overall it would be helpful for reproducibility if authors can visualize all the layers of all the different parts of the network as it is commonly done in the DL papers.
-	Talking about proposed formulation of BA use either of the following and be consistent across the paper:
Featuremetric BA / Feature-metric BA / Featuremetric BA / ‘Feature-metric BA’
-	Talking about depth parametrization use ‘basis’ or ‘bases’ not both and clearly defined the meaning of this important notion.
-	Attention should be given to the notation in formulas (3) and (4). The projection function there is no longer accepts a 3D point parametrized by 3 variables. Instead only depth is provided. 
In addition, the subindex ‘1’ of the point ‘q’ is not explained. 
-	More attention should be given to the evaluation section. Specifically to the tables (1 and 2) with quantitative results showing the comparison to other methods. 
It is not clear how the depth error is measured and it would be nicer to have the other errors explained exactly as they referred in the tables (e.g. ATE?).
-	How the first camera pose is initialized?
-	In Figure 2.b I’m surprised by the difference obtained in the feature maps for images which seems very similar (only the lighting seems to be different). Is it three consecutive frames?
-	Attention should be given to the grammar, formatting in particular the bibliography.