Paper:
Generating reliable illumination and viewpoint invariant keypoints is critical for tasks such as feature-based SLAM and SfM. Recently, several learned keypoint methods have demonstrated compelling performance on challenging benchmarks. However, generating consistent and accurate training data for interest-point detection in natural images still remains challenging, especially for human annotators. We introduce IO-Net (i.e. InlierOutlierNet), a novel proxy task for the self-supervision of keypoint detection, description and matching. By making the sampling of inlier-outlier sets from point-pair correspondences fully differentiable within the keypoint learning framework, we show that are able to simultaneously self-supervise keypoint description and improve keypoint matching. Second, we introduce KeyPointNet, a keypoint-network architecture that is especially amenable to robust keypoint detection and description. We design the network to allow local keypoint aggregation to avoid artifacts due to spatial discretizations commonly used for this task, and we improve fine-grained keypoint descriptor performance by taking advantage of efficient sub-pixel convolutions to upsample the descriptor feature-maps to a higher operating resolution. Through extensive experiments and ablative analysis, we show that the proposed self-supervised keypoint learning method greatly improves the quality of feature matching and homography estimation on challenging benchmarks over the state-of-the-art.

Review:
# UPDATE following rebuttal
Score increased to 6 due to architecture details in supplemental.
# Contributions
The paper contributes a self-supervised method of jointly learning 2D keypoint locations, descriptors, and scores given an input RGB image. The paper builds on previous work, adding:
* A more expressive keypoint location regression, which allows each 8x8 pixel region to vote for a keypoint location outside its boundary
* An upsampling step, similar to a U-net, to allow descriptors to be regressed with more detailed information
* An additional proxy task for the total loss, based on outlier rejection.
The authors train on COCO by manually distorting images to generate pairs with known homography, and show competitive results for keypoint detection and homography estimation tasks.
Decision: Weak reject. I would give this a 5 if the website allowed me to. A more detailed explanation of the neural network architecture, along with minor fixes described below, would make me increase my rating.
I feel the additions to existing pipelines are well motivated but insufficiently explained. In particular, the explanation of the neural network architecture along with figures 1 and 2 leaves many details unclear to me. Phrases like "a 1D CNN ... with 4 default setting residual blocks" is to me insufficient - residual networks have many details such as Resnet V1 or V2 style (ie is there a path right through the network which doesn't hit any activation functions), what kind of normalization is applied, number of channels in each block, how to do skips between different spatial resolutions, etc. The upsampling step for the descriptor head, which is claimed as a novel contribution, is not fully explained - "fast upsampling" implies (correctly) there are many variants of upsampling with different tradeoffs, but from the text I am unsure whether this is nearest neighbour upsampling, a ConvTranspose, etc. Similarly, "VGG style block" leaves some details unclear - whether the resolution downsampling is with a strided convolution / pooling / etc. Lots of the details are implied to be in other previous papers, but I feel that the paper would be hugely improved by exact architectural details.
There are various minor notational discrepancies in the paper - for example the outlier rejection is various defined as "InlierOuterNet (IONet)" and "The Inlier-Outlier model \emph{IO}", which also seems to be the same as the function $C$ defined a paragraph above. Perhaps it is common in this part of the literature, but to me an encoder decoder network is more likely to either be an autoencoder, or for the decoder to output something in the same modality (eg in machine translation). To say that some VGG blocks are an encoder, and the heads which produce keypoint locations / score / descriptor is a decoder, implies all neural networks could be described as an encoder/decoder.
The two figures showing the architecture are very different in design, which is not in itself a problem but the relationship between them could be clearer. I feel that the 'matching' box in figure 1 is misleading because it implies that matching only happens for the IONet, but the loss function for location described in Eq 1 also requires matching keypoints between the image pair. I'm also unclear on the division between direct and indirect supervisory signal - all the 4 loss components have a clear purpose, but it's not obvious what this partitioning means. "Indirect" only appears in this figure and the caption - perhaps.
The term "Anchor" appears only once with no reference, below equation 3 - I appreciate this is an existing term in this subfield, but given that the start of section 3 goes as far as explicitly defining what it means to produce 2D keypoints for an image, I feel defining this term would make the descriptor loss much clearer. 
One of the main contributions, that of allowing locations to regress outside their 8x8 area, sounds like a good idea but I feel that Figure 3 does not adequately show the benefit. In both a) and b), the blue estimates appear to be roughly as good as each other - clearly from the ablation a large benefit is gained from this innovation but perhaps a better illustrative example could be made here?
On a more positive note, I feel the components of the loss function are in general very clearly motivated and defined, and the description of training & data augmentation hyperparameters appears complete. If the description of the architecture could be improved that would result in a paper very amenable to reproduction.
The experiments are well explained, and the ablation of the various proposed  components is good. I feel table 1 would be improved with error bars - given that the bold best score is not exclusively next to V4, but in many cases the difference between V4 and the best is ~1%, error bars from different training runs might make clearer that V4 is overall the best configuration.
In the conclusion - "even without an explicit loss" - what is the difference between the loss functions used in this work, and an explicit loss?
Minor corrections:
The euclidean distance between descriptors is various notated as $d$ (section 3), $x$ (above equation 5) and $d$ again (below equation 5).
Typos: "normalzation" -> "normalization", "funcion" -> "function", "tripled" -> "triplet".