Paper:
Deep learning for object classification relies heavily on convolutional models. While effective, CNNs are rarely interpretable after the fact. An attention mechanism can be used to highlight the area of the image that the model focuses on thus offering a narrow view into the mechanism of classification. We expand on this idea by forcing the method to explicitly align images to be classified to reference images representing the classes. The mechanism of alignment is learned and therefore does not require that the reference objects are anything like those being classified. Beyond explanation, our exemplar based cross-alignment method enables classification with only a single example per category (one-shot). Our model cuts the 5-way, 1-shot error rate in Omniglot from 2.1% to 1.4% and in MiniImageNet from 53.5% to 46.5% while simultaneously providing point-wise alignment information providing some understanding on what the network is capturing. This method of alignment also enables the recognition of an unsupported class (open-set) in the one-shot setting while maintaining an F1-score of above 0.5 for Omniglot even with 19 other distracting classes while baselines completely fail to separate the open-set class in the one-shot setting.

Review:
Authors argue that using average (independent) greedy matching of pixel embedding (based on 4-6 layer cnn hypercolumns) is a better metric for one-shot learning than just using final layer embedding of a 4-6 layer cnn for the whole image.  Their argument is backed by outperforming their baseline and getting competitive results on few shot learning tasks. Their method is much more computationally heavy than the baseline matching networks. In order to make training feasible, in practice they train with 90% dropout of test pixels embedding & 80% dropout of reference pixels embedding. 
The caveats: 
-> Using hyper-columns is related to adding residual connections. The question remains how much performance can be gained by just adding residual connections (with dropout) to the matching networks and letting the network automatically (or with a probability) choose to embed higher layers or lower ones. Adding the residual connection and just comparing the final layer embeddings is a cleaner method than ABM which  provides a richer embedding than baseline and could potentially close the performance gap between ABM and final layer matching.
->It is strictly designed for one-shot learning. It does not benefit from few shots (extra shots) and the fact that these different shots are getting classified as the same label. Vinyal et al mitigates this shortcoming by adding the FCE. However FCE is not directly applicable anymore. Author’s don’t suggest any alternatives either. Their smaller gains (or even worse than baseline without self-regularization) in the 5-shot cases is an evidence of this shortcoming. 
The fact that SNAIL (TCML Mishra et al. (2017)) consistently outperforms this method puts a question mark on the significance of this work. If it was computationally feasible, authors could have used SNAIL and replaced the 64 dimensional embedding of each picture with the 10-20% hypercolumns. Essentially due to computational costs authors are sacrificing a more thorough matching system (non-greedy) for a richer embedding and they don’t get better results. 
On the other hand, authors may argue that the hyper-column matching is not just about performance, whereas it also adds interpretability to why two images are categorized the same. Illustrations like fig. 3 for example shows that the model is not matching semantically similar points and can be used to debug & improve the model. While understanding why a blackbox matching network is making a mistake and improving, is  harder. 
It would have been nice if authors used this added interpretability in some manner. Such as getting an idea about a regularizer, a prior, a mask, etc. and improved the performance.
I would argue for accepting this paper for two reasons.
-> Given that they beat their baseline and  they get comparable performance to sota even with a greedy matching (min-pooling followed by average pooling), is impressive. Furthermore, it is orthogonal to methods like SNAIL if the computational cost could be resolved.
-> They not only provide which image is a match but how they are matched, which could be interesting for one-shot detection as well as classification. 
Question: At test/validation: do you still only categorize with 10,20% samples or do you average the full attention map for all test pixels?
Nit: The manuscript needs several passes of proofreading, spell & grammar checking. A few examples in the first couple of pages:
-> The citing format needs to be fixed (like: LSTMsRavi, there should be () around citations). 
-> are not incompatible: are compatible
->incomprehensible sentence with two whiles: ABM networks outperforms these other state-of-the-art models on the open-set recognition task while in the one-shot setting by achieving a high accuracy of matching the non-open-set classes while maintaining a high F1 score for identifying samples in the open-set. 
-> add dots to the end of contribution list items.
-> we first look pairwise matchings: we first look at the pairwise matchings