Paper:
We study the interplay between memorization and generalization of overparameterized networks in the extreme case of a single training example and an identitymapping task. We examine fully-connected and convolutional networks (FCN and CNN), both linear and nonlinear, initialized randomly and then trained to minimize the reconstruction error. The trained networks stereotypically take one of two forms: the constant function (memorization) and the identity function (generalization). We formally characterize generalization in single-layer FCNs and CNNs. We show empirically that different architectures exhibit strikingly different inductive biases. For example, CNNs of up to 10 layers are able to generalize from a single example, whereas FCNs cannot learn the identity function reliably from 60k examples. Deeper CNNs often fail, but nonetheless do astonishing work to memorize the training output: because CNN biases are location invariant, the model must progressively grow an output pattern from the image boundaries via the coordination of many layers. Our work helps to quantify and visualize the sensitivity of inductive biases to architectural choices such as depth, kernel width, and number of channels.

Review:
This paper studies the inductive bias of neural nets by considering the toy example of learning an identity map through a single data point (and hence the NNs are always overparametrized). The authors compare CNNs versus FCNs, and find that CNNs tend to “generalize” in terms of actually learning the concept of an identity, whereas FCNs are prone to memorization. The authors also present results under various different settings such as changing the filter size or the number of hidden channels of CNNs. The conclusion is that the simpler the network architecture is, the better it generalizes. Another observation is that deep CNNs exhibit extreme memorization.
Overall, this is a well-written paper with an interesting set of experiments. However, I do have several concerns regarding the generality of the observed phenomenon in this paper. The first one is that the authors have chosen the comparison between CNNs and FCNs. In this case, I think a more fair comparison is to restrict the number of links (number of nonzero entries in the weight matrices) for both to be the same. In the current setup, however, the authors seem to consider the same number of hidden neurons, which naturally grants advantages to CNNs as they are of lower complexity.
Second, the training data is always a simple image from MNIST, and I am unsure how much this setting can generalize to other tasks, such as different data (say, music or text) or more complicated images (how do these experiments compare when we use a single data from ImageNet or CIFAR to train?). For instance, since CNNs are designed to capture invariances in natural images, it is unsurprising that they can generalize better on image data, but it would be quite astonishing if the same still holds true for acoustic data. In that case, the conclusion of this paper can be strengthened from “CNNs generalize better on image data” to “CNN generalize better”. Given the scope of the current paper, however, the best we can conclude is that “CNNs generalize better on MNIST”.
Last, again regarding the fair comparison, when comparing deep CNNs versus shallow ones, it is also of interest to see that, when restricted to the same number of parameters, if the deep CNNs still exhibit worse generalization. Otherwise, if some network complexities keep growing, we cannot really tell whether it is the network architecture that induces the inductive bias or it is simply the effect of complexities.
I also would like to suggest a future direction based on the idea in this paper: Comparing the inductive bias of GD versus SGD, a subject of intense study in the current literature. Since the authors considered a single training data, the results in this paper are always for GD. Now, say let us use 5 data, and compare the training of CNNs with different batch-size. Do the results differ? I think such a thought experiment would shed some light on the mysterious behaviors of the first-order algorithms that are widely used in practice.
Finally, a question: another submission to ICLR2020 [1] seems to suggest that optimization methods do not play a role in generalization, which is the opposite observation of this paper. Do the authors have any insight towards this contradiction?
[1] Fantastic Generalization Measures and Where to Find Them https://openreview.net/forum?id=SJgIPJBFvH