Paper:
LF-Net: Learning Local Features from Images
We present a novel deep architecture and a training strategy to learn a local feature pipeline from scratch, using collections of images without the need for human supervision. To do so we exploit depth and relative camera pose cues to create a virtual target that the network should achieve on one image, provided the outputs of the network for the other image. While this process is inherently non-differentiable, we show that we can optimize the network in a two-branch setup by confining it to one branch, while preserving differentiability in the other. We train our method on both indoor and outdoor datasets, with depth data from 3D sensors for the former, and depth estimates from an off-the-shelf Structure-from-Motion solution for the latter. Our models outperform the state of the art on sparse feature matching on both datasets, while running at 60+ fps for QVGA images.

Review:
Authors introduce a novel method for learning local feature detector and descriptors in a single framework. Compared to previous works, this method uses pair of images (instead of pair of image patches) and employs data with estimated depth maps and camera parameters. Learning on full images is achieved with an image loss which bypasses the lack of differentiability by computing gradient for only one branch. Even though the method is novel, I have concerns regarding the clarity and correctness of the evaluation.  The main strength of this work is the fact that it uses whole images for training the local feature detector. Thanks to the trick of using a global feature representation, authors are able to train with multiple images at once which mitigates the issue of an SGD update fitting a single image. Another claim, that the paper is able to discover valid points from scratch is unclear as it uses SFM results which, up to certain extent depend on quality of SIFT detector and descriptors. This makes their claim weaker and comparable to [1] which as well does not use an external local feature detector to bootstrap the training (and authors forget to cite this work).  However, my main concern is the lack of detail in the evaluation. As shown in many previous works, for evaluation of local feature matching algorithms, the measurement region size is a crucial parameter for a descriptor performance. Thus it is important to make sure that similar measurement regions size is used for all evaluated baselines in order to compare them fairly. A common approach is to evaluate the descriptor separately on pre-extracted patches [2, 3] which makes sure that the descriptors have access to the same information. And not all datasets have "heavily saturated scores". In this regard, it is not clear whether this has been taken into consideration. Similarly, it is not clear whether a similar amount of detected points was considered for each algorithm, as this can influence both the qualitative and quantitative evaluation.  However, a curse of local features is that each manuscript uses a different evaluation, thus a direct comparison is difficult. Separate evaluation of the detector a descriptors can yield some insight into the source of the main improvements, as e.g. the geometric precision of the detector is important for SFM application etc. Additionally, a speed comparison to other algorithms is missing. Comparison of performance on existing datasets would also be useful (such as in [4]), as it would allow to directly compare the numbers for the investigated baselines.   The writing does not seem to be well organized, mainly the Method section. The Figure 1 is hard to follow and explanation of branch i and j, used in section 3.2 is described in section 3.3. Similarly, it is not clear how the scores are combined across scales (Equation 1, across what elements is the sofmtax applied?), or how is scale taken into consideration while generating the "clean score maps", or how many points are considered for g. Similarly, it is not clear how the "split" mode of training works, considering that the orientation estimation is not being trained (line 203 and 273). It is not clear whether the common feature map is pre-trained or trained from scratch and how the camera parameters are used in the CNN (probably issue of wording).  Even though the method is quite interesting, my decision is mainly based on the issues with the evaluation which does not allow to verify fair comparison to existing methods. Additionally, I feel this submission would be more apt for computer vision conferences.  [1] Learning covariant feature detectors. K Lenc, A Vedaldi - Workshop of European Conference on Computer Vision, 2016. [2] Learning local image descriptors. S Winder and M Brown, CVPR 2007. [3] HPatches: A benchmark and evaluation of handcrafted and learned local descriptors, Balntas et al., CVPR 2017 [4] LIFT: Learned Invariant Feature Transform, Yi et al, ECCV 2016.  -------------------- Rebuttal feedback  I feel that the authors have addressed my main concerns really well and I am willing to improve the "overall score". It seems that the additional results provided in the rebuttal give additional evidence that the method is superior to existing ones and thus I would recommend the authors to include those in the final version of the paper together with improving the clarity of the text.  I am still slightly skeptical about their definition of the support region, provided as an average size of the region in pixels, which is a bit unconventional but ultimately seems to make sense. Traditionally, one uses magnification factor, however this actually addresses the issue of a possible bias in the detected scales.