Paper:
Deep convolutional network architectures are often assumed to guarantee generalization for small image translations and deformations. In this paper we show that modern CNNs (VGG16, ResNet50, and InceptionResNetV2) can drastically change their output when an image is translated in the image plane by a few pixels, and that this failure of generalization also happens with other realistic small image transformations. Furthermore, we see these failures to generalize more frequently in more modern networks. We show that these failures are related to the fact that the architecture of modern CNNs ignores the classical sampling theorem so that generalization is not guaranteed. We also show that biases in the statistics of commonly used image datasets makes it unlikely that CNNs will learn to be invariant to these transformations. Taken together our results suggest that the performance of CNNs in object recognition falls far short of the generalization capabilities of humans.

Review:
Paper summary: 
As is made clear in the title, this paper sets out to answer the following question: “Why do deep convolutional networks generalize so poorly to small image transformations?”. It focuses on natural image transformations on translation and scaling (rotation is missing though).
The paper proposes two main explanations: 
-	Strided convolution, called subsampling in the paper, ignores the classical sampling theorem,
-	CNNs will not learn invariance because of the (photographers') biases contained in the datasets.
On a general level, the paper is a good read, it is well written and the figures clearly convey the message they’re intended to. Adversarial attacks and robustness of CNNs in general is a very interesting and important topic in ML. The originality of this work is in the approach of the problem, the paper tries to explain the reasons why CNNs are vulnerable. Related works put more emphasis on coming up with novel attacks/defense strategies. Considering natural attacks as done in this submission is particularly interesting as it is probably a more surprising shortcoming of CNNs compared to optimally designed attacks or highly unnatural perturbations. The argument about subsampling (stride) being the reason of not having translational invariance is nice, especially the theoretical insight with the Shannon-Nyquist theorem and the more figurative example on part detectors. There are nevertheless a few major concerns about this work:
Major Concerns:
Theoretical arguments:
The theoretical argument made in this paper is interesting but to make the point stronger a more in-depth explanation would be needed.
-	The step from Eq (2) to Eq (3) is not entirely clear “K does not depend on x_i”, maybe one extra sentence to explain this step would be useful. 
-	Terms introduced such as the basis function B and the set of transformations T could be better defined.
-	For the extension to other types of transformations “While the claim focuses on global translation, it can also be extended to piecewise constant transformations.” it would be important to point out what type of natural transformations can be included in this set.
Empirical evidence:
Experiments are not fully convincing. Additional empirical evidence would be beneficial and necessary to support the claims of this:
-	“A natural criticism of these results is that they are somehow related to the image resizing and inpainting procedures that we used.” This is a very good point and the authors following arguments are not fully convincing. Results with different transformation procedures mentioned in the rest of the paragraph (and probably more) should be included to convince the reader.
-	The theoretical argument that translation invariance is not guaranteed because of the stride (subsampling) is not fully convincing and needs further explanation and experimental verification. In fact, feature maps of the CNNs that the authors consider do indeed contain many high frequencies.
-	The argument made in part 4 about the photographer’s bias seems valid for general natural transformations, but it does not apply to small transformations such as 1-pixel translations presented in the paper. Also, evidence that datasets without (or less) photographers' bias are less susceptible to natural attacks would make the argument in the paper a lot stronger. 
-	When using 6x6 avg pooling for the VGG16 architecture ”recognition performance decreases somewhat” . Results are only preliminary in the paper, but this statement needs a more thorough experimental backing. It should come with convincing quantitative evidence.
-	Please include some results or citation on other work about test time augmentation to support the statement “still only provides partial invariance”.
References and phrasing:
Generally previous work is well referenced in this paper, although there are some formulations that can be slightly modified to make a clear distinction between what is novel and what is previous work:
-	As is very well shown in the introduction, there is a lot of work on generating adversarial examples that drastically change the output of a CNN. This should be made clear in the abstract, in fact the sentence “In this paper we show that modern CNNs [...] also happens with other realistic small image transformations”  seems to indicate that this is the novel work in the paper. This is also why I believe the first sentence “Deep convolutional network architectures are often assumed to guarantee generalization for small image translations and deformations.” is somewhat contestable. 
-	“We find that modern deep CNNs are not invariant to translations, scalings and other realistic image transformations” as the paper points out earlier this is not a novel finding, so I would use a formulation that makes that clear and gives more emphasis to your own arguments as of why this happens.
Further Comments :
-	Part 5 "Implications for Practical Systems" could be moved to discussion as there is no new point and it seems more a reflection on what was already stated.
-	The final sentence of the abstract “Taken together our results suggest that the performance of CNNs in object recognition falls far short of the generalization capabilities of humans.” is not necessary, this is clearly true but it isn’t really contested in the ML community.
-	“despite the architecture being explicitly designed to provide such invariances” I agree that this has motivated the use and design of CNNs in the first place, but modern architectures are mostly designed to surpass the results on the common benchmarks rather than to provide such invariances.
-	”jaggedness is greater for the modern, deeper, networks compared to the less modern VGG16 network” might be worth interesting to consider if the residuals have anything to do with it.