Paper:
Memorization of data in deep neural networks has become a subject of significant research interest. In this paper, we link memorization of images in deep convolutional autoencoders to downsampling through strided convolution. To analyze this mechanism in a simpler setting, we train linear convolutional autoencoders and show that linear combinations of training data are stored as eigenvectors in the linear operator corresponding to the network when downsampling is used. On the other hand, networks without downsampling do not memorize training data. We provide further evidence that the same effect happens in nonlinear networks. Moreover, downsampling in nonlinear networks causes the model to not only memorize just linear combinations of images, but individual training images. Since convolutional autoencoder components are building blocks of deep convolutional networks, we envision that our findings will shed light on the important phenomenon of memorization in over-parameterized deep networks.

Review:
Summary:-
The authors investigates downsampling as one method by which autoencoding CNNs may memorize data. The theoretical motivation provided concentrates on linear CNNs. They show that downsampling linear CNNs tent to learn a point-map of the training data, even though (under certain initializations) they are capable of learning identity maps. However, non-downsampling linear CNNs learn identity maps. Given enough data however, the authors claim that the downsampling CNN will learn the identity map.
Strengths:-
+ Authors present a good exploration of how linear CNNs memorize data when they do downsampling. 
+ A theoretical prediction of the amount of training data needed to counteract data memorization for downsampling linear CNNs is provided, "Our conjecture also implies that when training a linear downsampling CNN on images of size 3 · 224 · 224, which corresponds to the input image size for VGG and ResNet (He et al. (2016), Simonyan & Zisserman (2015)), the number of linearly independent training examples needs to be at least 3 · 224 · 224 = 153, 228 before the network can learn the identity function." 
Weaknesses:-
+ Not enough theoretical proof is provided to support the hypothesis. Which would be fine but some key experiments are missing to make the paper empirically rigorous.
++ Would be good to see experiments that illustrate the predication that a certain amount of data would allow for learning identity maps, both for linear and non-linear CNNs.
++ In the non-linear CNN setting, I'd like to see the same early-stopping experiment done for linear CNNs whose results are in Fig. 3. I don't see any obvious theoretical reason why that result form Fig. 3 must extend to the non-linear setting. 
+ Initializations are pointed to as effecting the type of function the network learns. The authors give an example of a hand-designed initialization that allows a downsampling linear CNN to learn the identity map but they don't explain how they arrived at this initialization, or its properties that make it a good initialization. In general however, I think it's alright to assign exploration of effect of initialization to future work, since it seems like a non-trivial task.
+ It is mentioned that "the results are not observed for linear networks when using Kaiming initialization," which I read to mean the downsampling linear CNNs with Kaiming initialization learn the identity map, not point-map. If this is true, it seems like a vital point and should be included in discussions of future work.
Recommendation:  I think this could be a better short paper. There are some interesting contributions, but maybe not enough for a full length paper. For a full length paper, some further exploration of _why_ downsampling leads to (if indeed there is a causality) data memorization is needed.
Minor stuff:-
Citation "Gunasekar et al." is missing year (conclusions section)