Paper:
APPROXIMATING CNNS WITH BAG-OF-LOCAL- FEATURES MODELS WORKS SURPRISINGLY WELL ON IMAGENET
Deep Neural Networks (DNNs) excel on many complex perceptual tasks but it has proven notoriously difficult to understand how they reach their decisions. We here introduce a high-performance DNN architecture on ImageNet whose decisions are considerably easier to explain. Our model, a simple variant of the ResNet-50 architecture called BagNet, classifies an image based on the occurrences of small local image features without taking into account their spatial ordering. This strategy is closely related to the bag-of-feature (BoF) models popular before the onset of deep learning and reaches a surprisingly high accuracy on ImageNet (87.6% top-5 for 33 × 33 px features and Alexnet performance for 17 × 17 px features). The constraint on local features makes it straight-forward to analyse how exactly each part of the image influences the classification. Furthermore, the BagNets behave similar to state-of-the art deep neural networks such as VGG-16, ResNet-152 or DenseNet-169 in terms of feature sensitivity, error distribution and interactions between image parts. This suggests that the improvements of DNNs over previous bag-of-feature classifiers in the last few years is mostly achieved by better fine-tuning rather than by qualitatively different decision strategies.

Review:
This is an experimental paper that investigates how spatial ordering of patches influences the classification performances of CNNs. To do so, the authors design CNNs close to ResNets that almost only consist in a simple cascade of 1x1 convolutions, obtaining relatively small receptive field. It is an interesting read, and I recommend it as a valuable contribution to ICLR, that might lead to nice future works.
I have however several comments and questions, that I would like to be addressed.
1) First I think a reference is missing. Indeed, to my knowledge, it is not the first work to use this kind of techniques. Cf [1]. This does not alterate however the novelty of the approach.
2) « We construct a linear DNN-based BoF » : I do not like this formulation. Here, you assume that you build a ResNet-50 with 1x1 as a representation and have a last final linear layer as a classifier. One could also claim it is a ResNet-48 as a representation followed by 2 layers of 1x1 as a classifier.
3) « our proposed model architecture is simpler » this is very subjective because for instance the FV models are learned in a layer-wise fashion, which makes their learning procedure more interpretable because each layer objective is specified. Furthermore, analyzing these models is now equivalent to analyze a cascade of fully connected layers, which is not simple at all.
4) Again, the interpretability mentioned in Sec. 3  is in term of spatial localization, not mapping. I think it is important to make clear this consideration. Indeed, this work basically leaves the problem of understanding general CNNs to the problem of understanding MLPs.
5) The graphic of the Appendix A is a bit misleading : it seems 13 downsampling are performed whereas it is not the case, because the first element of each group of block is actually only done once.(if I understood correctly)
6) I think the word feature is sometimes mis-used: sometimes it seems it can refer to a patch, sometimes to the code for a patch. (« Surprisingly, feature sizes assmall as 17 × 17 pixels »)
I got also few questions:
Q1 : I was wondering if you did try manifold learning on the patches ? Do you expect it to work ?
Q2 : Is there a batch normalization in the FC or a normalization? Did you try to threshold the heat maps before feeding them to the linear layer? I'm wondering indeed if the amplitude of those heatmaps is really key.
Q3 : do you think it would be easy to exploit the non-overlapping patches for a better parallelization of computations ?
Finally, I find very positive the amount of experiments to test the similarity with standard CNNs. Of course, it’s far from being a formal proof, but I think it is a very nice first step.
[1] Oyallon, Edouard, Eugene Belilovsky, and Sergey Zagoruyko. "Scaling the scattering transform: Deep hybrid networks." International Conference on Computer Vision (ICCV). 2017.