Original: I DENTITY C RISIS : M EMORIZATION AND G ENERALIZATION UNDER E XTREME O VERPARAMETERIZATION
Revised: I DENTITY C RISIS : M EMORIZATION AND G ENERALIZATION UNDER E XTREME O VERPARAMETERIZATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We study the interplay between memorization and generalization of overparameterized networks in the extreme case of a single training example and an identitymapping task.
Revised: We study the interplay between memorization and generalization of overparameterized networks in the extreme case of a single training example and an identitymapping task.

Original: We examine fully-connected and convolutional networks (FCN and
Revised: We examine fully-connected and convolutional networks (FCN and

Original: CNN), both linear and nonlinear, initialized randomly and then trained to minimize the reconstruction error.
Revised: CNN), both linear and nonlinear, initialized randomly and then trained to minimize the reconstruction error.

Original: The trained networks stereotypically take one of two forms: the constant function ( memorization ) and the identity function ( generalization ).
Revised: The trained networks stereotypically take one of two forms: the constant function ( memorization ) and the identity function ( generalization ).

Original: We formally characterize generalization in single-layer FCNs and
Revised: We formally characterize generalization in single-layer FCNs and

Original: CNNs.
Revised: CNNs.

Original: We show empirically that different architectures exhibit strikingly different inductive biases.
Revised: We show empirically that different architectures exhibit strikingly different inductive biases.

Original: For example, CNNs of up to 10 layers are able to generalize from a single example whereas FCNs cannot learn the identity function reliably from 60k examples.
Revised: For example, CNNs of up to 10 layers are able to generalize from a single example, whereas FCNs cannot learn the identity function reliably from 60k examples.

Original: Deeper CNNs often fail, but nonetheless do astonishing work to memorize the training output: because CNN biases are location invariant, the model must progressively grow an output pattern from the image boundaries via the coordination of many layers.
Revised: Deeper CNNs often fail, but nonetheless do astonishing work to memorize the training output: because CNN biases are location invariant, the model must progressively grow an output pattern from the image boundaries via the coordination of many layers.

Original: Our work helps to quantify and visualize the sensitivity of inductive biases to architectural choices such as depth, kernel width, and number of channels.
Revised: Our work helps to quantify and visualize the sensitivity of inductive biases to architectural choices such as depth, kernel width, and number of channels.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The remarkable empirical success of deep neural networks is often attributed to the availability of large data sets for training.
Revised: The remarkable empirical success of deep neural networks is often attributed to the availability of large data sets for training.

Original: However, sample size does not provide a comprehensive rationale since complex models often outperform simple ones on a given data set, even when the model size exceeds the number of training examples.
Revised: However, sample size does not provide a comprehensive rationale since complex models often outperform simple ones on a given data set, even when the model size exceeds the number of training examples.

Original: What form of inductive bias leads to better generalization performance from highly overparameterized models?
Revised: What form of inductive bias leads to better generalization performance from highly overparameterized models?

Original: Numerous theoretical and empirical studies of inductive bias in deep learning have been conducted in recent years (Dziugaite & Roy, 2016;
Revised: Numerous theoretical and empirical studies of inductive bias in deep learning have been conducted in recent years (Dziugaite & Roy, 2016;

Original: Kawaguchi et al., 2017; Bartlett et al., 2017; Neyshabur et al., 2017; Liang et al., 2017; Neyshabur et al., 2018; Arora et al., 2018;
Revised: Kawaguchi et al., 2017; Bartlett et al., 2017; Neyshabur et al., 2017; Liang et al., 2017; Neyshabur et al., 2018; Arora et al., 2018;

Original: Zhou et al., 2019)
Revised: Zhou et al., 2019)

Original: but these postmortem analyses do not identify the root source of the bias.
Revised: but these postmortem analyses do not identify the root source of the bias.

Original: One cult belief among researchers is that gradient-based optimization methods provide an implicit bias toward simple solutions (Neyshabur et al., 2014;
Revised: One cult belief among researchers is that gradient-based optimization methods provide an implicit bias toward simple solutions (Neyshabur et al., 2014;

Original: Soudry et al., 2018; Shah et al., 2018; Arora et al., 2019).
Revised: Soudry et al., 2018; Shah et al., 2018; Arora et al., 2019).

Original: However, when a network is sufﬁciently large (e.g., the number of hidden units in each layer is polynomial in the input dimension and the number of training examples), then under some mild assumptions, gradient methods are guaranteed to ﬁt the training set perfectly (Allen-Zhu et al., 2018b; Du et al., 2018a;b; Zou et al., 2018).
Revised: However, when a network is sufﬁciently large (e.g., the number of hidden units in each layer is polynomial in the input dimension and the number of training examples), then under some mild assumptions, gradient methods are guaranteed to ﬁt the training set perfectly (Allen-Zhu et al., 2018b; Du et al., 2018a;b; Zou et al., 2018).

Original: These results do not distinguish a model trained on a data distribution with strong statistical regularities from one trained on the same inputs but with randomly shufﬂed labels.
Revised: These results do not distinguish a model trained on a data distribution with strong statistical regularities from one trained on the same inputs but with randomly shufﬂed labels.

Original: Although the former model might achieve good generalization, the latter can only memorize the training labels.
Revised: Although the former model might achieve good generalization, the latter can only memorize the training labels.

Original: Consequently, these analyses do not tell the whole story on the question of inductive bias.
Revised: Consequently, these analyses do not tell the whole story on the question of inductive bias.

Revised: [Figure]

Original: Another line of research characterizes sufﬁcient conditions on the input and label distribution that guarantee generalization from a trained network.
Revised: Another line of research characterizes sufﬁcient conditions on the input and label distribution that guarantee generalization from a trained network.

Original: These conditions range from linear separability (Brutzkus et al., 2018) to compact structures (Li & Liang, 2018).
Revised: These conditions range from linear separability (Brutzkus et al., 2018) to compact structures (Li & Liang, 2018).

Original: While very promising, this direction has thus far identiﬁed only structures that can be solved by linear or nearest neighbor classiﬁers over the original input space.
Revised: While very promising, this direction has thus far identiﬁed only structures that can be solved by linear or nearest neighbor classiﬁers over the original input space.

Original: The fact that in many applications deep neural networks signiﬁcantly outperform these simpler models reveals a gap in our understanding of deep neural networks.
Revised: The fact that in many applications deep neural networks signiﬁcantly outperform these simpler models reveals a gap in our understanding of deep neural networks.

Original: [Figure]

Original: As a formal understanding of inductive bias in deep networks has been elusive, we conduct a novel exploration in a highly restrictive setting that admits visualization and quantiﬁcation of inductive bias, allowing us to compare variations in architecture, optimization procedure, initialization scheme, and hyperparameters.
Revised: As a formal understanding of inductive bias in deep networks has been elusive, we conduct a novel exploration in a highly restrictive setting that admits visualization and quantiﬁcation of inductive bias, allowing us to compare variations in architecture, optimization procedure, initialization scheme, and hyperparameters.

Original: The particular task we investigate is learning an identity mapping in a regression setting.
Revised: The particular task we investigate is learning an identity mapping in a regression setting.

Original: The identity mapping is interesting for four reasons.
Revised: The identity mapping is interesting for four reasons.

Original: First, it imposes a structural regularity between the input and output, the type of regularity that could in principle lead to systematic generalization (He et al., 2016; Hardt & Ma, 2017).
Revised: First, it imposes a structural regularity between the input and output, the type of regularity that could in principle lead to systematic generalization (He et al., 2016; Hardt & Ma, 2017).

Original: Second, it requires that every input feature is transmitted to the output and thus provides a sensitive indicator of whether a model succeeds in passing activations (and gradients) between inputs and outputs.
Revised: Second, it requires that every input feature is transmitted to the output and thus provides a sensitive indicator of whether a model succeeds in passing activations (and gradients) between inputs and outputs.

Original: Third, conditional image generation is a popular task in the literature (e.g., Mirza & Osindero, 2014; Ledig et al., 2017); an identity mapping is the simplest form of such a generative process.
Revised: Third, conditional image generation is a popular task in the literature (e.g., Mirza & Osindero, 2014; Ledig et al., 2017); an identity mapping is the simplest form of such a generative process.

Original: Fourth, and perhaps most importantly, it admits detailed analysis and visualization of model behaviors and hidden representations.
Revised: Fourth, and perhaps most importantly, it admits detailed analysis and visualization of model behaviors and hidden representations.

Original: Consider networks trained on the identity task with 60k MNIST digits.
Revised: Consider networks trained on the identity task with 60k MNIST digits.

Original: Although only digit images are presented during the training, one might expect the strong regularity of the task to lead to good generalization to images other than digits.
Revised: Although only digit images are presented during the training, one might expect the strong regularity of the task to lead to good generalization to images other than digits.

Original: Figure 1 compares three different architectures.
Revised: Figure 1 compares three different architectures.

Original: The top row shows various input patterns, and the next three rows are outputs from a 20-layer convolutional net (CNN), a 10-layer fully connected net (FCN) with rectiﬁed-linear unit (ReLU) activation functions, and a 1-layer FCN.
Revised: The top row shows various input patterns, and the next three rows are outputs from a 20-layer convolutional net (CNN), a 10-layer fully connected net (FCN) with rectiﬁed-linear unit (ReLU) activation functions, and a 1-layer FCN.

Original: The latter amounts to a convex optimization problem with inﬁnitely many solutions, however gradient decent attains a unique closed-form solution.
Revised: The 1-layer FCN amounts to a convex optimization problem with inﬁnitely many solutions, however gradient decent converges to a unique closed-form solution.

Original: All nets perform well on the training set (ﬁrst three columns) and transfer well to novel digits and digit blends (columns 4–6).
Revised: All nets perform well on the training set (ﬁrst three columns) and transfer well to novel digits and digit blends (columns 4–6).

Original: Yet, outside of the hull of hand-printed digits, only the CNN discovers a reasonably good approximation to the identity function.
Revised: Yet, outside of the hull of hand-printed digits, only the CNN discovers a reasonably good approximation to the identity function.

Original: Figure 1 reﬂects architecture-speciﬁc inductive bias that persists even with 60k training examples.
Revised: Figure 1 reﬂects architecture-speciﬁc inductive bias that persists even with 60k training examples.

Original: Despite this persistence, a model’s intrinsic bias is more likely to be revealed with a smaller training set.
Revised: Despite this persistence, a model’s intrinsic bias is more likely to be revealed with a smaller training set.

Original: In this paper, we push this argument to the limit by studying learning with a single training example.
Revised: In this paper, we push this argument to the limit by studying learning with a single training example.

Original: Although models are free to reveal their natural proclivities in this maximally overparameterized regime, our initial intuition was that a single example would be uninteresting as models would be algebraically equivalent to the constant function (e.g., via biases on output units).
Revised: Although models are free to reveal their natural proclivities in this maximally overparameterized regime, our initial intuition was that a single example would be uninteresting as models would be algebraically equivalent to the constant function (e.g., via biases on output units).

Original: Further, it seemed inconceivable that inductive biases would be sufﬁciently strong to learn a mapping close to the identity.
Revised: Further, it seemed inconceivable that inductive biases would be sufﬁciently strong to learn a mapping close to the identity.

Original: Unexpectedly, our experiments show that model behavior is subtle and architecture dependent.
Revised: Unexpectedly, our experiments show that model behavior is subtle and architecture dependent.

Original: In a broad set of experiments, we highlight model characteristics—including depth, initialization, and hyperparameters—that determine where a model lands on the continuum between memorization (learning a constant function) and generalization (learning the identity function).
Revised: In a broad set of experiments, we highlight model characteristics—including depth, initialization, and hyperparameters—that determine where a model lands on the continuum between memorization (learning a constant function) and generalization (learning the identity function).

Original: The simplicity of the training scenario permits rich characterization of inductive biases.
Revised: The simplicity of the training scenario permits rich characterization of inductive biases.

Revised: [Figure]

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: The consequences of overparameterized models in deep learning have been extensively studied in recently years, on the optimization landscape and convergence of SGD (Allen-Zhu et al., 2018b;
Revised: The consequences of overparameterized models in deep learning have been extensively studied in recently years, on the optimization landscape and convergence of SGD (Allen-Zhu et al., 2018b;

Original: Du et al., 2018a;b; Bassily et al., 2018; Zou et al., 2018; Oymak & Soltanolkotabi, 2018), as well as the generalization guarantees under stronger structural assumptions of the data (Li & Liang, 2018; Brutzkus et al., 2018; Allen-Zhu et al., 2018a).
Revised: Du et al., 2018a;b; Bassily et al., 2018; Zou et al., 2018; Oymak & Soltanolkotabi, 2018), as well as the generalization guarantees under stronger structural assumptions of the data (Li & Liang, 2018; Brutzkus et al., 2018; Allen-Zhu et al., 2018a).

Original: Another line of related work is the study of the implicit regularization effects of SGD on training overparameterized models (Neyshabur et al., 2014; Zhang et al., 2017;
Revised: Another line of related work is the study of the implicit regularization effects of SGD on training overparameterized models (Neyshabur et al., 2014; Zhang et al., 2017;

Original: Soudry et al., 2018; Shah et al., 2018; Arora et al., 2019).
Revised: Soudry et al., 2018; Shah et al., 2018; Arora et al., 2019).

Original: [Figure]

Original: The traits of memorization in learning are also explicitly studied from various perspectives such as prioritizing learning of simple patterns (Arpit et al., 2017) or perfect interpolation of the training set (Belkin et al., 2018; Feldman, 2019).
Revised: The traits of memorization in learning are also explicitly studied from various perspectives such as prioritizing learning of simple patterns (Arpit et al., 2017) or perfect interpolation of the training set (Belkin et al., 2018; Feldman, 2019).

Original: More recently, coincidentally with the writing of this paper,
Revised: More recently, coincidentally with the writing of this paper,

Original: Radhakrishnan et al.
Revised: Radhakrishnan et al.

Original: (2018) reported on the effects of the downsampling operator in convolutional auto-encoders on image memorization.
Revised: (2018) reported on the effects of the downsampling operator in convolutional auto-encoders on image memorization.

Original: Their empirical framework is similar to ours, ﬁtting CNNs to the autoregression problem with few training examples.
Revised: Their empirical framework is similar to ours, ﬁtting CNNs to the autoregression problem with few training examples.

Original: We focus on investigating the general inductive bias in the extreme overparameterization case, and study a broader range of network types without enforcing a bottleneck in the architectures.
Revised: We focus on investigating the general inductive bias in the extreme overparameterization case, and study a broader range of network types without enforcing a bottleneck in the architectures.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We explore a progression of models: linear convex models, linear non-convex models (with multiple linear layers), fully-connected multilayered architectures with nonlinearities, and ﬁnally the case of greatest practical importance, fully convolutional networks.
Revised: We explore a progression of models: linear convex models, linear non-convex models (with multiple linear layers), fully-connected multilayered architectures with nonlinearities, and ﬁnally the case of greatest practical importance, fully convolutional networks.

Original: In all architectures we study, we ensure that there is a simple realization of the identity function (see Appendix B).
Revised: In all architectures we study, we ensure that there is a simple realization of the identity function (see Appendix B).

Original: We train networks by minimizing the mean squared error using standard gradient descent.
Revised: We train networks by minimizing the mean squared error using standard gradient descent.

Original: 3. F ULLY CONNECTED NETWORKS
Revised: 3. F ULLY CONNECTED NETWORKS

Original: Figure 2 shows examples of predictions from multi-layer fully connected networks.
Revised: Figure 2 shows examples of predictions from multi-layer fully connected networks.

Original: Inﬁnitely many solutions exist for all models under this extreme over-parameterization, and the ﬁgure shows that all the models ﬁt the training example perfectly.
Revised: Inﬁnitely many solutions exist for all models under this extreme over-parameterization, and the ﬁgure shows that all the models ﬁt the training example perfectly.

Original: However, on new test examples, contrasting behaviors are observed between shallow and deep networks.
Revised: However, on new test examples, contrasting behaviors are observed between shallow and deep networks.

Original: In particular, deeper models bias toward predicting a constant output, whereas shallower networks tend to predict random white noises on unseen inputs.
Revised: In particular, deeper models bias toward predicting a constant output, whereas shallower networks tend to predict random white noises on unseen inputs.

Original: The random predictions can be characterized as follows (proof in Appendix C).
Revised: The random predictions can be characterized as follows (proof in Appendix C).

Original: Theorem 1.
Revised: Theorem 1.

Original: A one-layer fully connected network, when trained with gradient descent on a single training example ˆ x , converges to a solution that makes the following prediction on a test example x :
Revised: A one-layer fully connected network, when trained with gradient descent on a single training example ˆ x , converges to a solution that makes the following prediction on a test example x :

Original: [Equation]
Revised: [Equation]

Original:  where x = Π (cid:107) ( x )+Π ⊥ ( x ) decomposes x into orthogonal components that are parallel and perpendicular to the training example ˆ x , respective.
Revised: [Figure] where x = Π (cid:107) ( x )+Π ⊥ ( x ) decomposes x into orthogonal components that are parallel and perpendicular to the training example ˆ x , respective.

Original: R is a random matrix from the network initialization, independent of the training data.
Revised: R is a random matrix from the network initialization, independent of the training data.

Original: For test examples similar to the training example—i.e., where Π (cid:107) ( x ) dominates Π ⊥ ( x ) —the outputs resemble the training output; on the other hand, for test examples that are not highly correlated to the training example, Π ⊥ ( x ) dominates and the outputs looks like white noise due to the random projection by R .
Revised: For test examples similar to the training example—i.e., where Π (cid:107) ( x ) dominates Π ⊥ ( x ) —the outputs resemble the training output; on the other hand, for test examples that are not highly correlated to the training example, Π ⊥ ( x ) dominates and the outputs looks like white noise due to the random projection by R .

Original: The behavior can be empirically veriﬁed from the second row in Figure 2.
Revised: The behavior can be empirically veriﬁed from the second row in Figure 2.

Original: Speciﬁcally, the ﬁrst test example is a mixture of the training and an unseen test example, and the[Figure] corresponding output is a mixture of white noise and the training output.
Revised: Speciﬁcally, the ﬁrst test example is a mixture of the training and an unseen test example, and the corresponding output is a mixture of white noise and the training output.

Original: For the remaining test examples, the outputs appear random.
Revised: For the remaining test examples, the outputs appear random.

Original: Although Theorem 1 characterizes only the 1-layer linear case, the empirical results in Figuresuggest that shallow (2 layer) networks tend to have this inductive bias.
Revised: Although Theorem 1 characterizes only the 1-layer linear case, the empirical results in Figuresuggest that shallow (2 layer) networks tend to have this inductive bias.

Original: However, this inductive bias does not miraculously obtain good generalization: the trained model fails to learn either the identity or the constant function.
Revised: However, this inductive bias does not miraculously obtain good generalization: the trained model fails to learn either the identity or the constant function.

Original: Speciﬁcally, it predicts well in the vicinity (measured by correlations) of the training example ˆ x , but further away its predictions are random.
Revised: Speciﬁcally, it predicts well in the vicinity (measured by correlations) of the training example ˆ x , but further away its predictions are random.

Original: In particular, when the test example x is orthogonal to ˆ x , the prediction is completely random.
Revised: In particular, when the test example x is orthogonal to ˆ x , the prediction is completely random.

Original: In deeper (6 layer) nets, the deviations from the identity function take on a quite different characteristic form.
Revised: In deeper (6 layer) nets, the deviations from the identity function take on a quite different characteristic form.

Original: Interestingly, deeper linear networks behave more like deeper ReLU networks, with a strong bias towards a constant function that maps any input to the single training output.
Revised: Interestingly, deeper linear networks behave more like deeper ReLU networks, with a strong bias towards a constant function that maps any input to the single training output.

Original: A multilayer linear network with no hidden-layer bottleneck has essentially the same representational power as a 1-layer linear network, but gradient descent produces different learning dynamics that alter the inductive biases.
Revised: A multilayer linear network with no hidden-layer bottleneck has essentially the same representational power as a 1-layer linear network, but gradient descent produces different learning dynamics that alter the inductive biases.

Original: See Appendix E for more results and analysis on FCNs.
Revised: See Appendix E for more results and analysis on FCNs.

Revised: [Figure]

Original: 3. C ONVOLUTIONAL NETWORKS
Revised: 3. C ONVOLUTIONAL NETWORKS

Original: We next study the inductive bias of convolutional neural networks with ReLU activation functions.
Revised: We next study the inductive bias of convolutional neural networks with ReLU activation functions.

Original: Figure 3 shows predictions on various test patterns obtained by training CNNs of varying depths.
Revised: Figure 3 shows predictions on various test patterns obtained by training CNNs of varying depths.

Original: Compared to FCNs, CNNs have strong structural constraints that limit the receptive ﬁeld of each neuron to a spatially local neighborhood and the weights are tied and being used across the spatial array.
Revised: Compared to FCNs, CNNs have strong structural constraints that limit the receptive ﬁeld of each neuron to a spatially local neighborhood and the weights are tied and being used across the spatial array.

Original: These two constraints match the structure of the identity target function.
Revised: These two constraints match the structure of the identity target function.

Original: (See Appendix B. for an example of constructing the identity function with CNNs.)
Revised: (See Appendix B. for an example of constructing the identity function with CNNs.)

Original: Similar to the fully connected case, for one-layer CNN, we can bound the error as follows (proof in Appendix D).[Figure]
Revised: Similar to the fully connected case, for one-layer CNN, we can bound the error as follows (proof in Appendix D).

Original: Theorem 2.
Revised: Theorem 2.

Original: A one-layer convolutional neural network can learn the identity map from a single training example with the mean squared error over all output pixels bounded as
Revised: A one-layer convolutional neural network can learn the identity map from a single training example with the mean squared error over all output pixels bounded as

Original: [Equation]
Revised: [Equation]

Original: where m is the number of network parameters, C is the number of channels in the image, and r ≤ m / C is the rank of the subspace formed by the span of the local input patches.
Revised: where m is the number of network parameters, C is the number of channels in the image, and r ≤ m / C is the rank of the subspace formed by the span of the local input patches.

Original: The error grows with m , the number of parameters in the network.
Revised: The error grows with m , the number of parameters in the network.

Original: For example, learning CNNs with larger receptive ﬁeld sizes will be harder.
Revised: For example, learning CNNs with larger receptive ﬁeld sizes will be harder.

Original: Even though the bound seems to decrease with more (input and output) channels in the image, note that the number of channels C also contributes to the number of parameters ( m = K H K W C 2 ), so there is a trade-off.
Revised: Even though the bound seems to decrease with more (input and output) channels in the image, note that the number of channels C also contributes to the number of parameters ( m = K H K W C 2 ), so there is a trade-off.

Original: Unlike typical generalization bound that decays with number of i.i.d.
Revised: Unlike typical generalization bound that decays with number of i.i.d.

Original: training examples, we have only one training example here, and the key quantity that reduces the bound is the rank r of the subspace formed by the local image patches.
Revised: training examples, we have only one training example here, and the key quantity that reduces the bound is the rank r of the subspace formed by the local image patches.

Original: The size of the training image implicitly affects bounds as larger image generates more image patches.
Revised: The size of the training image implicitly affects bounds as larger image generates more image patches.

Original: Note the rank r also heavily depends on the contents of the training image.
Revised: Note the rank r also heavily depends on the contents of the training image.

Original: For example, simply padding the image with zeros on all boundaries will not reduce the error bound.
Revised: For example, simply padding the image with zeros on all boundaries will not reduce the error bound.

Original: With enough linearly independent image patches, the subspace becomes full rank r
Revised: With enough linearly independent image patches, the subspace becomes full rank r

Original: = m / C , and learning of the global identity map is guaranteed.
Revised: = m / C , and learning of the global identity map is guaranteed.

Original: The theorem guarantees only the one-layer case.
Revised: The theorem guarantees only the one-layer case.

Original: Empirically—as shown in Figure 3—CNNs with depth up-to-5 layers learn a fairly accurate approximation to the identity function, with the exception of a few artifacts at the boundaries.
Revised: Empirically—as shown in Figure 3—CNNs with depth up-to-5 layers learn a fairly accurate approximation to the identity function, with the exception of a few artifacts at the boundaries.

Original: For a quantitative evaluation, we measure the performance by calculating the correlation (See Appendix J for the results in MSE.) to two reference functions: the identity function and the constant function that maps every input to the training point ˆ x .
Revised: For a quantitative evaluation, we measure the performance by calculating the correlation (See Appendix J for the results in MSE.) to two reference functions: the identity function and the constant function that maps every input to the training point ˆ x .

Original: To examine how a model’s response varies with similarity to the training image, we generate test images having correlation ρ ∈ [0 , 1] to the training image by: (1) sampling an image with random pixels, (2) adding α ˆ x to the image, picking α such that the correlation with ˆ x is ρ ; (3) renormalizing the image to be of the same norm as ˆ x .
Revised: To examine how a model’s response varies with similarity to the training image, we generate test images having correlation ρ ∈ [0 , 1] to the training image by: (1) sampling an image with random pixels, (2) adding α ˆ x to the image, picking α such that the correlation with ˆ x is ρ ; (3) renormalizing the image to be of the same norm as ˆ x .

Original: For ρ = 0 , the test images are orthogonal to ˆ x , whereas for ρ = 1 , the test images equal ˆ x .
Revised: For ρ = 0 , the test images are orthogonal to ˆ x , whereas for ρ = 1 , the test images equal ˆ x .

Original: The results for CNNs of different depths are shown in Figure 4.
Revised: The results for CNNs of different depths are shown in Figure 4.

Original: The quantitative ﬁndings are consistent with the visualizations: shallow CNNs are able to learn the identity function from only one training example; very deep CNNs bias towards the constant function; and CNNs of intermediate depth correlate well with neither the identity nor the constant function.
Revised: The quantitative ﬁndings are consistent with the visualizations: shallow CNNs are able to learn the identity function from only one training example; very deep CNNs bias towards the constant function; and CNNs of intermediate depth correlate well with neither the identity nor the constant function.

Original: However, unlike FCNs that produce white-noise-like predictions, from Figure 3 CNNs of intermediate depth behave like edge detectors.
Revised: However, [Figure] E rr o r r a t e layers

Original: To evaluate how much information is lost in the intermediate layers, we use the following simple criterion to assess the representation in each layer.
Revised: To evaluate how much information is lost in the intermediate layers, we use the following simple criterion to assess the representation in each layer.

Original: We feed each image in the MNIST dataset through a network trained on our single example.
Revised: We feed each image in the MNIST dataset through a network trained on our single example.

Original: We collect the representations at a given layer and perform a simple similarity-weighted classiﬁcation.
Revised: We collect the representations at a given layer and perform a simple similarity-weighted classiﬁcation.

Original: For each example x i from the MNIST test set, we predict its class as a weighted average of the (one hot) label vector of each example x j from the
Revised: For each example x i from the MNIST test set, we predict its class as a weighted average of the (one hot) label vector of each example x j from the

Original: MNIST training set, where the weight is the inner-product of x i and x j .
Revised: MNIST training set, where the weight is the inner-product of x i and x j .

Original: [Figure] E rr o r r a t e layers This metric does not quantify how much information is preserved as the image representation propagates through the layers, because the representation could still be maintained yet not captured by the simple correlation-weighted classiﬁer.
Revised: This metric does not quantify how much information is preserved as the image representation propagates through the layers, because the representation could still be maintained yet not captured by the simple correlation-weighted classiﬁer.

Original: It nonetheless provides a simple metric for exploring the (identity) mapping: using the input image as the baseline, if a layer represents the identity function, then the representation at that layer would obtain a similar error rate when using the input representation; on the other hand, if a layer degenerates into the constant function, then the corresponding representation would have error rate close to random guessing of the label.
Revised: It nonetheless provides a simple metric for exploring the (identity) mapping: using the input image as the baseline, if a layer represents the identity function, then the representation at that layer would obtain a similar error rate when using the input representation; on the other hand, if a layer degenerates into the constant function, then the corresponding representation would have error rate close to random guessing of the label.

Original: The results are plotted in Figure 5.
Revised: The results are plotted in Figure 5.

Original: The error curve for a randomly initialized 20-layer CNN is shown as reference: at random initialization, the smoothing effect renders the representations beyond the sixth layer unuseful for the averaging-based classiﬁer.
Revised: The error curve for a randomly initialized 20-layer CNN is shown as reference: at random initialization, the smoothing effect renders the representations beyond the sixth layer unuseful for the averaging-based classiﬁer.

Original: After training, the concave nonmonotonicity in the curves indicates loss and then recovery of the information present in the input.
Revised: After training, the concave nonmonotonicity in the curves indicates loss and then recovery of the information present in the input.

Original: Trained networks try to recover the washed out intermediate layer representations as means to link the input and the output layer.
Revised: Trained networks try to recover the washed out intermediate layer representations as means to link the input and the output layer.

Original: However, if the depth is too large, the network tries to infer input-output relations using partial information, resulting in models that behave like edge detectors.
Revised: However, if the depth is too large, the network tries to infer input-output relations using partial information, resulting in models that behave like edge detectors.

Original: Finally, for the case of 20 layers, the curve shows that the bottom few layers do get small improvements in error rate comparing to random initialization, but the big gap between the input and output layer drives the network to learn the constant function instead.
Revised: Finally, for the case of 20 layers, the curve shows that the bottom few layers do get small improvements in error rate comparing to random initialization, but the big gap between the input and output layer drives the network to learn the constant function instead.

Original: On a ﬁrst sight, this deems to underscore a vanishing gradient problem, but Figure 1 reveals that given a sufﬁcient number of training examples, a 20-layer CNN cancan still learn the identity map.
Revised: On a ﬁrst sight, this deems to underscore a vanishing gradient problem, but Figure 1 reveals that given a sufﬁcient number of training examples, a 20-layer CNN can still learn the identity map.

Original: See also Appendix G for further details.
Revised: See also Appendix G for further discussions on vanishing gradients.

Original: Since CNNs preserve spatial structure, we can also visualize information loss in the intermediate layers.
Revised: Since CNNs preserve spatial structure, we can also visualize information loss in the intermediate layers.

Original: The visualization results, described in Appendix F, are consistent with the aforementioned observations.
Revised: The visualization results, described in Appendix F, are consistent with the aforementioned observations.

Original: 3. R OBUSTNESS TO CHANGES IN INPUT SCALE
Revised: 3. R OBUSTNESS TO CHANGES IN INPUT SCALE

Original: Whether a relatively shallow CNN learns the identity function from a single training example or a relatively deep CNN learns the constant function, both outcomes reﬂect an inductive bias because the training objective never explicitly mandates the model to learn one structure or another.
Revised: Whether a relatively shallow CNN learns the identity function from a single training example or a relatively deep CNN learns the constant function, both outcomes reﬂect an inductive bias because the training objective never explicitly mandates the model to learn one structure or another.

Original: The spatial structure of CNNs enables additional analyses of the encoding induced by the learned function.
Revised: The spatial structure of CNNs enables additional analyses of the encoding induced by the learned function.

Original: In particular, we examined how changing the size of the input by scaling the dimensions of the spatial map affects the model predictions.
Revised: In particular, we examined how changing the size of the input by scaling the dimensions of the spatial map affects the model predictions.

Original: Figure 6 depicts the predictions of a 5-layer CNN trained on a 28 × 28 image and tested on 7 × 7 and 112 × 112 images.
Revised: Figure 6 depicts the predictions of a 5-layer CNN trained on a 28 × 28 image and tested on 7 × 7 and 112 × 112 images.

Original: Although the learned identity map generally holds up against a larger-than-trained input, the identity map is disturbed on smaller-thantrained inputs.
Revised: Although the learned identity map generally holds up against a larger-than-trained input, the identity map is disturbed on smaller-thantrained inputs.

Original: See Appendix H.1 for a comprehensive description of the results.
Revised: See Appendix H.1 for a comprehensive description of the results.

Original: Note that CNNs are capable of encoding the identity function for arbitrary input and ﬁlter sizes (see Appendix B.3).
Revised: Note that CNNs are capable of encoding the identity function for arbitrary input and ﬁlter sizes (Appendix B.3).

Original: Figure 7 shows the predictions on the rescaled input patterns of Figure 6 by a 20-layer CNN that has learned the constant function on a 28 × 28 image.
Revised: Figure 7 shows the predictions on the rescaled input patterns of Figure 6 by a 20-layer CNN that has learned the constant function on a 28 × 28 image.

Original: The learned constant map holds up over a smaller range of input sizes than the learned identity map in a 5-layer CNN.
Revised: The learned constant map holds up over a smaller range of input sizes than the learned identity map in a 5-layer CNN.

Original: It is nonetheless interesting to see smooth changes as the input size increases to reveal the network’s own notion of “7”, clearly deﬁned with respect to the corners of the input map (see Figure 21 in Appendix H for how the patterns progressively grow from image boundaries in intermediate layers).
Revised: It is nonetheless interesting to see smooth changes as the input size increases to reveal the network’s own notion of “7”, clearly deﬁned with respect to the corners of the input map (Figure 21 in Appendix H reveals interesting details on how the patterns progressively grow from image boundaries in intermediate layers).

Original: We performed additional experiments to directly feed test images to the upper subnet, which reveals more about the generative process by which the net synthesizes a constant output (Appendix H).
Revised: We performed additional experiments to directly feed test images to the upper subnet, which reveals more about the generative process by which the net synthesizes a constant output (Appendix H).

Original: 3. V ARYING OTHER FACTORS DURING TRAINING
Revised: 3. V ARYING OTHER FACTORS DURING TRAINING

Original: We studied a variety of common hyperparameters, including image dimensions, convolutional ﬁlter dimensions, number of kernels, weight initialization scheme, and the choice of gradient-based optimizer.
Revised: We studied a variety of common hyperparameters, including image dimensions, convolutional ﬁlter dimensions, number of kernels, weight initialization scheme, and the choice of gradient-based optimizer.

Original: With highly overparameterized networks, training converges to zero error for a wide range of hyperparameters.
Revised: With highly overparameterized networks, training converges to zero error for a wide range of hyperparameters.

Original: Within the models that all perform optimally on the training set, we now address how the particular hyperparameter settings affect the inductive bias.
Revised: Within the models that all perform optimally on the training set, we now address how the particular hyperparameter settings affect the inductive bias.

Original: We brieﬂy present some of the most interesting observations here; please refer to Appendix I for the full results and analyses.
Revised: We brieﬂy present some of the most interesting observations here; please refer to Appendix I for the full results and analyses.

Original: Size of training image.
Revised: Size of training image.

Original: Figure 8 shows, for varying training-image size, the mean correlation to the constant and the identity function at different depths within the network.
Revised: Figure 8 shows, for varying training-image size, the mean correlation to the constant and the identity function at different depths within the network.

Original: The training examples are resized versions of the same image.
Revised: The training examples are resized versions of the same image.

Original: The bias toward the constant function at a given depth increases with smaller training images.
Revised: The bias toward the constant function at a given depth increases with smaller training images.

Original: This ﬁnding makes sense considering that smaller images provide fewer pixel-to-pixel mapping constraints, which is aligned with Theorem 2.
Revised: This ﬁnding makes sense considering that smaller images provide fewer pixel-to-pixel mapping constraints, which is aligned with Theorem 2.

Original: Size of convolution ﬁlters.
Revised: Size of convolution ﬁlters.

Original: Figure 9 illustrates the inductive bias with convolution ﬁlter size varying from 5 × 5 to 57 × 57 .
Revised: Figure 9 illustrates the inductive bias with convolution ﬁlter size varying from 5 × 5 to 57 × 57 .

Original: Predictions become blurrier as the ﬁlter size grows.
Revised: Predictions become blurrier as the ﬁlter size grows.

Original: With extremely large ﬁlters that cover the entire input array, CNNs exhibit a strong bias towards the constant function.
Revised: With extremely large ﬁlters that cover the entire input array, CNNs exhibit a strong bias towards the constant function.

Original:  Because the training inputs are of size 28 × 28 , a 29 × 29 ﬁlter size allows each neuron to see at least half of the spatial domain of the previous layer, assuming large boundary padding of the inputs.
Revised: [Figure] [Figure] [Figure] Because the training inputs are of size 28 × 28 , a 29 × 29 ﬁlter size allows each neuron to see at least half of the spatial domain of the previous layer, assuming large boundary padding of the inputs.

Original: With 57 × 57 ﬁlters centered at any location within the image, each neuron sees the entire previous layer.
Revised: With 57 × 57 ﬁlters centered at any location within the image, each neuron sees the entire previous layer.

Original: The error boundin Theorem 2 also deteriorates as the ﬁlter sizes increases.
Revised: This is also consistent with Theorem 2, in which the error bound deteriorates as the ﬁlter sizes increases.

Original: Note even with a large ﬁlter, CNNs do not perform the same elementary computation as FCNs because the  convolution ﬁlter is repeatedly applied throughout the spatial domain.
Revised: Note even with a large ﬁlter, CNNs do not perform the same elementary computation as FCNs because the (shared) convolution ﬁlter is repeatedly applied throughout the spatial domain.

Original: Number of channels.
Revised: Number of channels.

Original: Appendix B.3 shows that in principle two channels sufﬁce to encode the identity function for gray-scale inputs via a straightforward construction.
Revised: Appendix B.3 shows that in principle two channels sufﬁce to encode the identity function for gray-scale inputs via a straightforward construction.

Original: However, in practice the outcome of training may be quite different depending on the channel count, as shown in Figure 10(a) [Figure]
Revised: However, in practice the outcome of training may be quite different depending on the channel count, as shown in Figure 10(a).

Original: [Figure] [Figure] On the one hand, the aggressively overparameterized network with 1024 channels ( ∼ 25M parameters per middle convolution layer) does not seem to suffer from overﬁtting.
Revised: On the one hand, the aggressively overparameterized network with 1024 channels ( ∼ 25M parameters per middle convolution layer) does not seem to suffer from overﬁtting.

Original: On the other hand, the results with three channels often lose content in the image center.
Revised: On the other hand, the results with three channels often lose content in the image center.

Original: The problem is not underﬁtting as the network reconstructs the training image (ﬁrst column) correctly.
Revised: The problem is not underﬁtting as the network reconstructs the training image (ﬁrst column) correctly.

Original: A potential reason why 3-channel CNNs do not learn the identity map well might be poor initialization when there are very few channels (Frankle & Carbin, 2019).
Revised: A potential reason why 3-channel CNNs do not learn the identity map well might be poor initialization when there are very few channels (Frankle & Carbin, 2019).

Original: This issue is demonstrated in Figure 29 of
Revised: This issue is demonstrated in Figure 29 of

Original: Appendix I.
Revised: Appendix I.

Original: Our study of training with different initialization schemes indeed conﬁrms that the random initial conditions have a big impact on the inductive bias (Appendix I).
Revised: Our study of training with different initialization schemes indeed conﬁrms that the random initial conditions have a big impact on the inductive bias (Appendix I).

Original: Figure 10(b) shows the case for 20-layer CNNs.
Revised: Figure 10(b) shows the case for 20-layer CNNs.

Original: Surprisingly, having an order of magnitude more feature channels in every layer does not seem to help much at making the information ﬂow through layers, as the network still learns to ignore the inputs and construct a constant output.
Revised: Surprisingly, having an order of magnitude more feature channels in every layer does not seem to help much at making the information ﬂow through layers, as the network still learns to ignore the inputs and construct a constant output.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We presented an systematic study of the extreme case of overparameterization when learning from a single example.
Revised: We presented an systematic study of the extreme case of overparameterization when learning from a single example.

Original: We investigated the interplay between memorization and generalization in deep neural networks.
Revised: We investigated the interplay between memorization and generalization in deep neural networks.

Original: By restricting the learning task to the identity function, we sidestepped issues such as the underlying optimal Bayes error of the problem and the approximation error of the hypothesis classes.
Revised: By restricting the learning task to the identity function, we sidestepped issues such as the underlying optimal Bayes error of the problem and the approximation error of the hypothesis classes.

Original: This choice also facilitated rich visualization and intuitive interpretation of the trained models.
Revised: This choice also facilitated rich visualization and intuitive interpretation of the trained models.

Original: Under this setup, we investigated gradient-based learning procedures with explicit memorization-generalization characterization.
Revised: Under this setup, we investigated gradient-based learning procedures with explicit memorization-generalization characterization.

Original: Our results indicate that different architectures exhibit vastly different inductive bias towards memorization and generalization.
Revised: Our results indicate that different architectures exhibit vastly different inductive bias towards memorization and generalization.

Original: For future work, we plan to extend the study to other domains and neural network architectures, like natural language processing and recurrent neural networks, and aim for more qualitative relationship between the inductive bias and various architecture conﬁgurations. 
Revised: For future work, we plan to extend the study to other domains and neural network architectures, like natural language processing and recurrent neural networks and aim for more qualitative relationship between the inductive bias and various architecture conﬁgurations. A CKNOWLEDGMENTS

Revised: We would like to thank Kunal Talwar, Hanie Sedghi and Rong Ge for helpful discussions

Original: A E XPERIMENT DETAILS AND HYPER - PARAMETERS
Revised: A E XPERIMENT DETAILS AND HYPER - PARAMETERS

Original: We specify the experiment setups and the hyper-parameters here.
Revised: We specify the experiment setups and the hyper-parameters here.

Original: Unless otherwise speciﬁed in each study (e.g. when we explicitly vary the number of convolution channels), all the hyper-parameters are set according to the default values here.
Revised: Unless otherwise speciﬁed in each study (e.g. when we explicitly vary the number of convolution channels), all the hyper-parameters are set according to the default values here.

Original: The main study is done with the MNIST dataset.
Revised: The main study is done with the MNIST dataset.

Original: It consists of grayscale images of hand written digits of size 28 × 28 .
Revised: It consists of grayscale images of hand written digits of size 28 × 28 .

Original: For training, we randomly sample one digit from the training set (a digit ‘7’) with a ﬁxed random seed.
Revised: For training, we randomly sample one digit from the training set (a digit ‘7’) with a ﬁxed random seed.

Original: For testing, we use random images from the test set of MNIST and
Revised: For testing, we use random images from the test set of MNIST and

Original: Fashion-MNIST, as well as algorithmically generated structured patterns and random images.
Revised: Fashion-MNIST, as well as algorithmically generated structured patterns and random images.

Original: The training and test images are all normalized by mapping the pixel values in { 0 , 1 , . . .
Revised: The training and test images are all normalized by mapping the pixel values in { 0 , 1 , . . .

Original: , 255 } to [0 , 1] , and then standardize with the mean 0 .
Revised: , 255 } to [0 , 1] , and then standardize with the mean 0 .

Original: 1307 and standard deviation 0 .
Revised: 1307 and standard deviation 0 .

Original: 3081 originally calculated on the (full) MNIST training set.
Revised: 3081 originally calculated on the (full) MNIST training set.

Original: The models are trained by minimizing the mean squared error (MSE) loss with a vanilla SGD (base learning rate 0.01 and momentum 0.9).
Revised: The models are trained by minimizing the mean squared error (MSE) loss with a vanilla SGD (base learning rate 0.01 and momentum 0.9).

Original: The learning rate is scheduled as stagewise constant that decays with a factor of 0.2 at the 30%, 60% and 80% of the total training steps (2,000,000).
Revised: The learning rate is scheduled as stagewise constant that decays with a factor of 0.2 at the 30%, 60% and 80% of the total training steps (2,000,000).

Original: No weight decay is applied during training.
Revised: No weight decay is applied during training.

Original: For neural network architectures, the rectiﬁed linear unit (ReLU) activation is used for both fully connected networks (FCNs) and convolutional networks (CNNs).
Revised: For neural network architectures, the rectiﬁed linear unit (ReLU) activation is used for both fully connected networks (FCNs) and convolutional networks (CNNs).

Original: The input and output dimensions are decided by the data.
Revised: The input and output dimensions are decided by the data.

Original: The hidden dimensions for the FCNs are 2,048 by default.
Revised: The hidden dimensions for the FCNs are 2,048 by default.

Original: The CNNs use 5 × 5 kernels with stride 1 and padding 2, so that the geometry does not change after each convolution layer.
Revised: The CNNs use 5 × 5 kernels with stride 1 and padding 2, so that the geometry does not change after each convolution layer.

Original: B R EPRESENTATION OF THE IDENTITY FUNCTION USING DEEP NETWORKS
Revised: B R EPRESENTATION OF THE IDENTITY FUNCTION USING DEEP NETWORKS

Original: In this section, we provide explicit constructions on how common types of neural networks can represent the identity function.
Revised: In this section, we provide explicit constructions on how common types of neural networks can represent the identity function.

Original: Those constructions are only proof for that the models in our study have the capacity to represent the target function.
Revised: Those constructions are only proof for that the models in our study have the capacity to represent the target function.

Original: There are many different ways to construct the identity map for each network architecture, but we try to provide the most straightforward and explicit constructions.
Revised: There are many different ways to construct the identity map for each network architecture, but we try to provide the most straightforward and explicit constructions.

Original: However, during our experiments, even when the SGD learns (approximately) the identity function, there is no evidence suggesting that it is encoding the functions in similar ways as described here.
Revised: However, during our experiments, even when the SGD learns (approximately) the identity function, there is no evidence suggesting that it is encoding the functions in similar ways as described here.

Original: We put some mild constraints (e.g. no “bottleneck” in the hidden dimensions) to allow more straightforward realization of the identity function, but this by no means asserts that networks violating those constraints cannot encode the identity function.
Revised: We put some mild constraints (e.g. no “bottleneck” in the hidden dimensions) to allow more straightforward realization of the identity function, but this by no means asserts that networks violating those constraints cannot encode the identity function.

Original: B. L INEAR MODELS
Revised: B. L INEAR MODELS

Original: For a one-layer linear network f ( x ) =
Revised: For a one-layer linear network f ( x ) =

Original: Wx , where W ∈ R d × d , setting W to the identity matrix will realize the identity function.
Revised: Wx , where W ∈ R d × d , setting W to the identity matrix will realize the identity function.

Original: For a multi-layer linear network f ( x ) =
Revised: For a multi-layer linear network f ( x ) =

Original: ( (cid:81) (cid:96) W (cid:96) )
Revised: ( (cid:81) (cid:96) W (cid:96) )

Original: x , we need to require that all the hidden dimensions are not smaller than the input dimension.
Revised: x , we need to require that all the hidden dimensions are not smaller than the input dimension.

Original: In this case, a simple concrete construction is to set each W (cid:96) to an identity matrix.
Revised: In this case, a simple concrete construction is to set each W (cid:96) to an identity matrix.

Original: B. M ULTI - LAYER R E LU NETWORKS
Revised: B. M ULTI - LAYER R E LU NETWORKS

Original: The ReLU activation function σ ( · ) = max(0 , · ) discards all the negative values.
Revised: The ReLU activation function σ ( · ) = max(0 , · ) discards all the negative values.

Original: There are many ways one can encode the negative values and recover it after ReLU.
Revised: There are many ways one can encode the negative values and recover it after ReLU.

Original: We provide a simple approach that uses hidden dimensions twice the input dimension.
Revised: We provide a simple approach that uses hidden dimensions twice the input dimension.

Original: Consider a ReLU network with one hidden layer f ( x )
Revised: Consider a ReLU network with one hidden layer f ( x )

Original: =
Revised: =

Original: W 2 σ ( W 1 x ) , where W 2 ∈ R d × 2 d , W 1 ∈ R 2 d × d .
Revised: W 2 σ ( W 1 x ) , where W 2 ∈ R d × 2 d , W 1 ∈ R 2 d × d .

Original: The idea is to store the positive and negative part of x separately, and then re-construct.
Revised: The idea is to store the positive and negative part of x separately, and then re-construct.

Original: This can be achieved by setting
Revised: This can be achieved by setting

Original: [-W 1 = (cid:18) I d − I d (cid:19) , W 
-]
Revised: [-W 1 = (cid:18) I d − I d (cid:19) , W 
-][Equation]

Original: = ( I d − I d ) where I d is the d -dimensional identity matrix.
Revised: where I d is the d -dimensional identity matrix.

Original: For the case of more than two layers, we can use the bottom layer to split the positive and negative part, and the top layer to merge them back.
Revised: For the case of more than two layers, we can use the bottom layer to split the positive and negative part, and the top layer to merge them back.

Original: All the intermediate layers can be set to 2 d -dimensional identity matrix.
Revised: All the intermediate layers can be set to 2 d -dimensional identity matrix.

Original: Since the bottom layer encode all the responsives in non-negative values, the ReLU in the middle layers will pass through.
Revised: Since the bottom layer encode all the responsives in non-negative values, the ReLU in the middle layers will pass through.

Original: B. C ONVOLUTIONAL NETWORKS
Revised: B. C ONVOLUTIONAL NETWORKS

Original: In particular, we consider 2D convolutional networks for data with the structure of multi-channel images.
Revised: In particular, we consider 2D convolutional networks for data with the structure of multi-channel images.

Original: A mini-batch of data is usually formatted as a four-dimensional tensor of the shape
Revised: A mini-batch of data is usually formatted as a four-dimensional tensor of the shape

Original: B × C × H × W , where B is the batch size, C the number of channels (e.g. RGB or feature channels for intermediate layer representations), H and W are image height and width, respectively.
Revised: B × C × H × W , where B is the batch size, C the number of channels (e.g. RGB or feature channels for intermediate layer representations), H and W are image height and width, respectively.

Original: A convolutional layer (ignoring the bias term) is parameterized with another four-dimensional tensor of the shape ¯ C × C × K H
Revised: A convolutional layer (ignoring the bias term) is parameterized with another four-dimensional tensor of the shape ¯ C × C × K H

Original: × K W , where ¯ C is the number of output feature channels, K H and
Revised: × K W , where ¯ C is the number of output feature channels, K H and

Original: K W are convolutional kernel height and width, respectively.
Revised: K W are convolutional kernel height and width, respectively.

Original: The convolutional kernel is applied at local K H
Revised: The convolutional kernel is applied at local K H

Original: × K W patches of the input tensor, with optional padding and striding.
Revised: × K W patches of the input tensor, with optional padding and striding.

Original: For one convolution layer to represent the identity function, we can use only the center slice of the kernel tensor and set all the other values to zero.
Revised: For one convolution layer to represent the identity function, we can use only the center slice of the kernel tensor and set all the other values to zero.

Original: Note it is very rare to use even numbers as kernel size, in which case the “center” of the kernel tensor is not well deﬁned.
Revised: Note it is very rare to use even numbers as kernel size, in which case the “center” of the kernel tensor is not well deﬁned.

Original: When the kernel size is odd, we can set
Revised: When the kernel size is odd, we can set

Original: [Equation]
Revised: [Equation]

Original: By using only the center of the kernel, we essentially simulate a 1 × 1 convolution, and encode a local identity function for each (multi-channel) pixel.
Revised: By using only the center of the kernel, we essentially simulate a 1 × 1 convolution, and encode a local identity function for each (multi-channel) pixel.

Original: For multi-layer convolutional networks with ReLU activation functions, the same idea as in multilayer fully-connected networks can be applied.
Revised: For multi-layer convolutional networks with ReLU activation functions, the same idea as in multilayer fully-connected networks can be applied.

Original: Speciﬁcally, we ask for twice as many channels as the input channels for the hidden layers.
Revised: Speciﬁcally, we ask for twice as many channels as the input channels for the hidden layers.

Original: At the bottom layer, separately the positive and negative part of the inputs, and reconstruct them at the top layer.
Revised: At the bottom layer, separately the positive and negative part of the inputs, and reconstruct them at the top layer.

Original: C P ROOF OF T HEOREM
Revised: C P ROOF OF T HEOREM

Original: Consider the 1-layer linear model f W ( x ) =
Revised: Consider the 1-layer linear model f W ( x ) =

Original: Wx , where x ∈ R d and W ∈ R d × d .
Revised: Wx , where x ∈ R d and W ∈ R d × d .

Original: Let ˆ x be the single training example.
Revised: Let ˆ x be the single training example.

Original: The training objective is to minimize the empirical risk 
Revised: The training objective is to minimize the empirical risk ˆ R =

Original: ˆ R = 1 / 2 (cid:107) f W
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: x
Revised: (cid:107) f W

Original: [Equation]
Revised: [Equation]

Original: The optimization problem is convex and well understood.
Revised: The optimization problem is convex and well understood.

Original: Due to overparameterization, the solution of the empirical risk minimization is not unique.
Revised: Due to overparameterization, the solution of the empirical risk minimization is not unique.

Original: However, given randomly initialized weights W 0 , gradient descent obtains a unique global minimizer.
Revised: However, given randomly initialized weights W 0 , gradient descent obtains a unique global minimizer.

Original: The gradient of the empirical risk is
Revised: The gradient of the empirical risk is

Original: [Equation]
Revised: [Equation]

Original: Gradient descent with step sizes η t and initialization weights W 0 updates weights as
Revised: Gradient descent with step sizes η t and initialization weights W 0 updates weights as

Original: [Equation]
Revised: [Equation]

Original: where u T ∈ R
Revised: where u T ∈ R

Original: d is a vector decided via the accumulation in the optimization trajectory.
Revised: d is a vector decided via the accumulation in the optimization trajectory.

Original: Because of the form of the gradient, it is easy to see the solution found by gradient descent will always have such parameterization structure.
Revised: Because of the form of the gradient, it is easy to see the solution found by gradient descent will always have such parameterization structure.

Original: Moreover, under this parameterization, a unique minimizer exists that solves the equation
Revised: Moreover, under this parameterization, a unique minimizer exists that solves the equation

Original: [Equation]
Revised: [Equation]

Original: via
Revised: via

Original: [Equation]
Revised: [Equation]

Original: Therefore, the global minimizer can be written as
Revised: Therefore, the global minimizer can be written as

Original: [Equation]
Revised: [Equation]

Original: For the one-layer network case, the optimization problem is convex.
Revised: For the one-layer network case, the optimization problem is convex.

Original: Under standard conditions in convex optimization, gradient descent will converge to the global minimizer shown above.
Revised: Under standard conditions in convex optimization, gradient descent will converge to the global minimizer shown above.

Original: We can easily verify that the red term is exactly the projection of x onto the training example ˆ x , while the blue term is the residual projection onto the orthogonal subspace.
Revised: We can easily verify that the red term is exactly the projection of x onto the training example ˆ x , while the blue term is the residual projection onto the orthogonal subspace.

Original: D P ROOF OF T HEOREM
Revised: D P ROOF OF T HEOREM

Original: Lemma 1.
Revised: Lemma 1.

Original: Consider the linear model f ( x ) =
Revised: Consider the linear model f ( x ) =

Original: W (cid:62) x , where x ∈ R D , W ∈ R D × d .
Revised: W (cid:62) x , where x ∈ R D , W ∈ R D × d .

Original: Let the training set be { ( x 1 , y 1 ) , . . .
Revised: Let the training set be { ( x 1 , y 1 ) , . . .

Original: ( x N , y N ) } .
Revised: ( x N , y N ) } .

Original: Assume the model is overparameterized ( N ≤ d ) and there is no sample redundancy (rank of the data matrix is N ).
Revised: Assume the model is overparameterized ( N ≤ d ) and there is no sample redundancy (rank of the data matrix is N ).

Original: Fitting f
Revised: Fitting f

Original: ( · ) by optimizing the square loss with gradient descent on the training set converges to ˆ f ( x ) = ˆ W (cid:62) x , where
Revised: ( · ) by optimizing the square loss with gradient descent on the training set converges to ˆ f ( x ) = ˆ W (cid:62) x , where

Original: [Equation]
Revised: [Equation]

Original: W 0 ∈ R D ×
Revised: W 0 ∈ R D ×

Original: d is the random initialization of weights, X = ( x 1 , . . .
Revised: d is the random initialization of weights, X = ( x 1 , . . .

Original: , x N ) (cid:62) ∈ R N × D and Y =
Revised: , x N ) (cid:62) ∈ R N × D and Y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y N ) (cid:62) ∈ R N × d are the matrices formed by the inputs and outputs, respectively.
Revised: , y N ) (cid:62) ∈ R N × d are the matrices formed by the inputs and outputs, respectively.

Original: Lemma 2.
Revised: Lemma 2.

Original: With the same notation of Lemma 1, assume the model is underparameterized ( N > d ) and there is no feature redundancy (rank of the data matrix is d ).
Revised: With the same notation of Lemma 1, assume the model is underparameterized ( N > d ) and there is no feature redundancy (rank of the data matrix is d ).

Original: Fitting f
Revised: Fitting f

Original: ( · ) by optimizing the square loss with gradient descent on the training set converges to ˆ f ( x ) = ˆ W (cid:62) x where 
Revised: ( · ) by optimizing the square loss with gradient descent on the training set converges to ˆ f ( x ) = ˆ W (cid:62) x where [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: Let ˆ x ∈ R
Revised: Let ˆ x ∈ R

Original: H
Revised: H

Original: × W × C be the single training image of size H × W , and C color channels.
Revised: × W × C be the single training image of size H × W , and C color channels.

Original: Let K H
Revised: Let K H

Original: × K W be the convolution receptive ﬁeld size, so the weights of convolution ﬁlters can be parameterized via a 4-dimensional tensor Θ ∈ R K H
Revised: × K W be the convolution receptive ﬁeld size, so the weights of convolution ﬁlters can be parameterized via a 4-dimensional tensor Θ ∈ R K H

Original: × K W × C × C .
Revised: × K W × C × C .

Original: Let Ξ be the collection of 2D coordinates of the local patches from the input image that the convolutional ﬁlter is applied to, and P ij (ˆ x ) ∈ R K H
Revised: Let Ξ be the collection of 2D coordinates of the local patches from the input image that the convolutional ﬁlter is applied to, and P ij (ˆ x ) ∈ R K H

Original: × K W × C be the local patch of ˆ x centered at the coordinate ( i, j ) ∈ Ξ .
Revised: × K W × C be the local patch of ˆ x centered at the coordinate ( i, j ) ∈ Ξ .

Original: Note for our case the input and output are of the same shape, so the convolution stride is one, and | Ξ | = H × W .
Revised: Note for our case the input and output are of the same shape, so the convolution stride is one, and | Ξ | = H × W .

Original: The empirical risk of ﬁtting ˆ x with a one-layer convolution model can be written as
Revised: The empirical risk of ﬁtting ˆ x with a one-layer convolution model can be written as

Original: [Equation]
Revised: [Equation]

Original: where Θ ::: k ∈ R K H × K W × C is the subset of convolution weights corresponding to the k -th output channel, and ˆ x ijk is the pixel value at coordinate ( i, j ) and channel k .
Revised: where Θ ::: k ∈ R K H × K W × C is the subset of convolution weights corresponding to the k -th output channel, and ˆ x ijk is the pixel value at coordinate ( i, j ) and channel k .

Original: C ASE 1 — overparameterization: | Ξ | ≤ K H
Revised: C ASE 1 — overparameterization: | Ξ | ≤ K H

Original: × K W × C × C .
Revised: × K W × C × C .

Original: Assumme the patches are linearly independent 1 , with slight abuse of notatons, we represent the empirical risk in matrix form with theWhen patches i,
Revised: Assumme the patches are linearly independent 1 , with slight abuse of notatons, we represent the empirical risk in matrix form with the

Original: Ξ which be

Original: [Equation]
Revised: [Equation]

Original: can ﬁnd a linearly independent basis for the patches and rewrite the ﬁtting objectives with the basis.
Revised: ﬁnd a linearly independent basis for the patches and rewrite the ﬁtting objectives with the basis.

Original: following matrices 2 :
Revised: following matrices 2 :

Original: [Equation]
Revised: [Equation]

Original: and apply Lemma 1 to obtain ˆ W .
Revised: and apply Lemma 1 to obtain ˆ W .

Original: For a test example x , the prediction of the ( i, j ) -th (multi-channel) pixel is
Revised: For a test example x , the prediction of the ( i, j ) -th (multi-channel) pixel is

Original: [Equation]
Revised: [Equation]

Original: Note we are learning the identity map, the ( i, j ) -th row of the learning target matrix
Revised: Note we are learning the identity map, the ( i, j ) -th row of the learning target matrix

Original: Y is the (multi-channel) pixel at the ( i, j ) -th coordinate of ˆ x .
Revised: Y is the (multi-channel) pixel at the ( i, j ) -th coordinate of ˆ x .

Original: This is exactly the center of the ( i, j ) -th patch P ij (ˆ x ) , i.e. the ( i, j ) -th row of the matrix X .
Revised: This is exactly the center of the ( i, j ) -th patch P ij (ˆ x ) , i.e. the ( i, j ) -th row of the matrix X .

Original: As a result, there is a linear projection matrix Λ ∈ R ( K H K W C )
Revised: As a result, there is a linear projection matrix Λ ∈ R ( K H K W C )

Original: × C that maps X to Y : X Λ = Y .
Revised: × C that maps X to Y : X Λ = Y .

Original: So
Revised: So

Original: [Equation]
Revised: [Equation]

Original: where Π (cid:107) X
Revised: where Π [Equation]

Original:  = X
Revised: X = X

Original: [Equation]
Revised: [Equation]

Original: XX
Revised: XX

Original: [Equation]
Revised: [Equation]

Original: ) − 1 X is the linear operator that projects a vector into the subspace spanned by the rows of X , and Π ⊥ X
Revised: ) − 1 X is the linear operator that projects a vector into the subspace spanned by the rows of X , and Π ⊥ X

Original: = I − Π (cid:107) X is the operator projecting into the orthogonal subspace.
Revised: = I − Π (cid:107) X is the operator projecting into the orthogonal subspace.

Original: To compute the prediction errors, it is sufﬁce to look at the errors at each ( i, j ) -th
Revised: To compute the prediction errors, it is sufﬁce to look at the errors at each ( i, j ) -th

Original: (multi-channel) pixel separately:
Revised: (multi-channel) pixel separately:

Original: [Equation]
Revised: [Equation]

Original: Assume the absolute value of pixel values are bounded by B , then
Revised: Assume the absolute value of pixel values are bounded by B , then

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst two terms can be bounded according to the rank r of the subspace projection Π (cid:107) X .
Revised: The ﬁrst two terms can be bounded according to the rank r of the subspace projection Π (cid:107) X .

Original: Let the nullity of the projection be n = K H K W C − r .
Revised: Let the nullity of the projection be n = K H K W C − r .

Original: Note the projection matrix can be decomposed as
Revised: Note the projection matrix can be decomposed as

Original: [Equation]
Revised: [Equation]

Original: where { v k } k is a orthonormal basis for the projection subspace.
Revised: where { v k } k is a orthonormal basis for the projection subspace.

Original: [Equation]
Revised: [Equation]

Original: Similarly,
Revised: Similarly,

Original: [Equation]
Revised: [Equation]

Original: Assume the entries of W 0 are initialized as i.i.d.
Revised: Assume the entries of W 0 are initialized as i.i.d.

Original: Gaussians N (0 , σ 2 ) .
Revised: Gaussians N (0 , σ 2 ) .

Original: Since { v k } k are orthonormal vectors, for each k , let ρ k = (cid:107) W 0 (cid:62) v k (cid:107) 2 /σ 2 , then ρ k is distributed according to χ 2 distribution with
Revised: Since { v k } k are orthonormal vectors, for each k , let ρ k = (cid:107) W 0 (cid:62) v k (cid:107) 2 /σ 2 , then ρ k is distributed according to χ 2 distribution with

Original: C degree of freedom.
Revised: C degree of freedom.

Original: For any 1 < ζ ≤ M , where M is a constant chosen a priori, and for each k = 1 , . . . , n ,
Revised: For any 1 < ζ ≤ M , where M is a constant chosen a priori, and for each k = 1 , . . . , n ,

Original: [Equation]
Revised: [Equation]

Original: Markov’s inequality)
Revised: Markov’s inequality)

Original: [Equation]
Revised: [Equation]

Original: By union bound,
Revised: By union bound,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Let δ equals the right hand side, and solve for ζ , we get for any δ ≥ δ 0 > 0 , with probability at least
Revised: Let δ equals the right hand side, and solve for ζ , we get for any δ ≥ δ 0 > 0 , with probability at least

Original: [Equation]
Revised: [Equation]

Original: [Equation] where δ 0 ≥ 2 n / ( C ( M − log M − 1)) is chosen to satisfy ζ ≤ M .
Revised: 11) where δ 0 ≥ 2 n / ( C ( M − log M − 1)) is chosen to satisfy ζ ≤ M .

Original: We can also choose δ 0 ﬁrst and set M accordingly.
Revised: We can also choose δ 0 ﬁrst and set M accordingly.

Original: Putting everyting together, with probability at least 1 − δ , the mean squared error (averaged over all output pixels)
Revised: Putting everyting together, with probability at least 1 − δ , the mean squared error (averaged over all output pixels)

Original: [Equation]
Revised: [Equation]

Original: Note K H K W C 2 is the number of parameters in the convolution net.
Revised: Note K H K W C 2 is the number of parameters in the convolution net.

Original: C ASE 2 — underparameterization: | Ξ | > K H
Revised: C ASE 2 — underparameterization: | Ξ | > K H

Original: × K W × C × C .
Revised: × K W × C × C .

Original: Using the same notation above, assuming no redundant features, we apply Lemma 2 to get the prediction of the ( i, j ) -th (multichannel) pixel of a test example x as
Revised: Using the same notation above, assuming no redundant features, we apply Lemma 2 to get the prediction of the ( i, j ) -th (multichannel) pixel of a test example x as

Original: [Equation]
Revised: [Equation]

Original: Recall the deﬁnition of Λ , which maps a patch to the corresponding (multi-channel) pixel.
Revised: Recall the deﬁnition of Λ , which maps a patch to the corresponding (multi-channel) pixel.

Original: In this case, the prediction is exact, and the error is zero.
Revised: In this case, the prediction is exact, and the error is zero.

Original: Since in this case n , this case can be merged with equation 12.
Revised: Since in this case n , this case can be merged with equation 12.

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: The proof is an extension of Theorem 1 to the case of more than one training examples.
Revised: The proof is an extension of Theorem 1 to the case of more than one training examples.

Original: Using the notation in the Lemma, the training objective can be written as the matrix form
Revised: Using the notation in the Lemma, the training objective can be written as the matrix form

Original: [Equation]
Revised: [Equation]

Original: and the gradient as
Revised: and the gradient as

Original: [Equation]
Revised: [Equation]

Original: Since the model is overparameterized, and there is no unique minimizer to the empirical risk.
Revised: Since the model is overparameterized, and there is no unique minimizer to the empirical risk.

Original: However, gradient descent converges to a unique solution.
Revised: However, gradient descent converges to a unique solution.

Original: Note the gradient descent step is:
Revised: Note the gradient descent step is:

Original: [Equation]
Revised: [Equation]

Original: where U t = − (cid:80) t − 1 τ =1 η τ ( XW τ − Y ) ∈
Revised: where U t = − (cid:80) t − 1 τ =1 η τ ( XW τ − Y ) ∈

Original: R N ×
Revised: R N ×

Original: d parameterizes the solution at iteration t .
Revised: d parameterizes the solution at iteration t .

Original: Since XX (cid:62) is invertible in this case.
Revised: Since XX (cid:62) is invertible in this case.

Original: A unique solution exists under this parameterization, which can be obtained by solving
Revised: A unique solution exists under this parameterization, which can be obtained by solving

Original: [Equation]
Revised: [Equation]

Original: Plug this into the parameterization, we get
Revised: Plug this into the parameterization, we get

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: Using the same notation as in the proof of Lemma 1, since the model is underparameterized, a unique minimizer of the empirical risk exists.
Revised: Using the same notation as in the proof of Lemma 1, since the model is underparameterized, a unique minimizer of the empirical risk exists.

Original: Directly solving for the optimality condition ∂
Revised: Directly solving for the optimality condition ∂

Original: ˆ R/∂W = 0 , we get
Revised: ˆ R/∂W = 0 , we get

Original: [Equation]
Revised: [Equation]

Original: E F ULL RESULTS OF FULLY CONNECTED MULTI - LAYER NETWORKS
Revised: E F ULL RESULTS OF FULLY CONNECTED MULTI - LAYER NETWORKS

Original: In this section, we present the detailed results on fully connected networks that are omitted from
Revised: In this section, we present the detailed results on fully connected networks that are omitted from

Original: Section 3.1 due to space limit.
Revised: Section 3.1 due to space limit.

Original: E. F ULLY CONNECTED LINEAR NETWORKS
Revised: E. F ULLY CONNECTED LINEAR NETWORKS

Original: Figure 11 shows the results on multi-layer linear networks with various number of hidden layers and hidden units.
Revised: Figure 11 shows the results on multi-layer linear networks with various number of hidden layers and hidden units.

Original: The depth of the architecture has a stronger effect on the inductive bias than the width.
Revised: The depth of the architecture has a stronger effect on the inductive bias than the width.

Original: For example, the network with one hidden layer of dimension 2048 has 3.2M parameters, more than the 2.5M parameters of the network with three hidden layers of dimension 784.
Revised: For example, the network with one hidden layer of dimension 2048 has 3.2M parameters, more than the 2.5M parameters of the network with three hidden layers of dimension 784.

Original: But the latter behaves less like the convex case.
Revised: But the latter behaves less like the convex case.

Original: E. T WO - LAYER FULLY CONNECTED R E LU NETWORKS
Revised: E. T WO - LAYER FULLY CONNECTED R E LU NETWORKS

Original: Li & Liang (2018) offer a theoretical characterization of learning in a two-layer ReLU neural network.
Revised: Li & Liang (2018) offer a theoretical characterization of learning in a two-layer ReLU neural network.

Original: They show that when the data consists of well separated clusters (i.e., the cluster diameters are much smaller than the distances between each cluster pair), training an overparameterized twolayer ReLU network will generalize well.
Revised: They show that when the data consists of well separated clusters (i.e., the cluster diameters are much smaller than the distances between each cluster pair), training an overparameterized twolayer ReLU network will generalize well.

Original: To simplify the analysis, they study a special case where the weights in the top layer are randomly initialized and ﬁxed; only the bottom layer weights are learned.
Revised: To simplify the analysis, they study a special case where the weights in the top layer are randomly initialized and ﬁxed; only the bottom layer weights are learned.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We study the problem of learning a two-layer ReLU network under our identity-mapping task.
Revised: We study the problem of learning a two-layer ReLU network under our identity-mapping task.

Original: Figure 12 compares the cases of learning the bottom layer only and learning both layers (left and right panels, respectively).
Revised: Figure 12 compares the cases of learning the bottom layer only and learning both layers (left and right panels, respectively).

Original: The two cases demonstrate different inductive biases for predictions on unseen test images.
Revised: The two cases demonstrate different inductive biases for predictions on unseen test images.

Original: When only the ﬁrst layer is trained, the tendency is toward speckled noise, but when both layers are trained, the tendency is toward a constant output (the image used for training).
Revised: When only the ﬁrst layer is trained, the tendency is toward speckled noise, but when both layers are trained, the tendency is toward a constant output (the image used for training).

Original: Our observation does not contradict the theoretical results of Li & Liang (2018), which assume a well separated and clustered data distribution.
Revised: Our observation does not contradict the theoretical results of Li & Liang (2018), which assume a well separated and clustered data distribution.

Original: For bottom-layer only training, we can analyze it in a similar way to the one-layer networks.
Revised: For bottom-layer only training, we can analyze it in a similar way to the one-layer networks.

Original: Let us denote
Revised: Let us denote

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (15) where W ∈ R m × d is the learnable weight matrix, and α ∈ R m is randomly initialized and ﬁxed.
Revised: where W ∈ R m × d is the learnable weight matrix, and α ∈ R m is randomly initialized and ﬁxed.

Original: Although the trained weights no longer have a closed-form solution, the solution found by gradient descent is always parameterized as
Revised: Although the trained weights no longer have a closed-form solution, the solution found by gradient descent is always parameterized as

Original: [Equation]
Revised: [Equation]

Original: (16) where ˆ x is the training example, and u T ∈ R m summarizes the efforts of gradient descent up to time
Revised: (16) where ˆ x is the training example, and u T ∈ R m summarizes the efforts of gradient descent up to time

Original: T .
Revised: T .

Original: In particular, the gradient of the empirical risk ˆ R with respect to each row W : r of the learnable weight is
Revised: In particular, the gradient of the empirical risk ˆ R with respect to each row W : r of the learnable weight is

Original: [Equation]
Revised: [Equation]

Original: Putting it together, the full gradient is
Revised: Putting it together, the full gradient is

Original: [Equation]
Revised: [Equation]

Original: Since the gradient lives in the span of the training example ˆ x , the solution found by gradient descent is always parameterized as equation 16.
Revised: Since the gradient lives in the span of the training example ˆ x , the solution found by gradient descent is always parameterized as equation 16.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ) shows the results for 
Revised: function; (b) shows the results for linear FCNs.

Original: The same arguments applies to multi-layer neural networks.
Revised: The same arguments applies to multi-layer neural networks.

Original: The prediction on any test example that is orthogonal to ˆ x will depend only on randomly initialized W 0 and upper layer weights.
Revised: The prediction on any test example that is orthogonal to ˆ x will depend only on randomly initialized W 0 and upper layer weights.

Original: When only the bottom layer is trained, the upper layer weights will also be independent from the data, therefore the prediction is completely random.
Revised: When only the bottom layer is trained, the upper layer weights will also be independent from the data, therefore the prediction is completely random.

Original: However, when all the layers are jointly trained, the arguments no longer apply.
Revised: However, when all the layers are jointly trained, the arguments no longer apply.

Original: The empirical results presented in the main text that multi-layer networks bias towards the constant function verify this.
Revised: The empirical results presented in the main text that multi-layer networks bias towards the constant function verify this.

Original: In particular, if the test example is orthogonal to ˆ x (i.e., ˆ x (cid:62) x = 0 ), the prediction depends solely on the randomly initialized values in W 0 and therefore can be characterized by the distribution used for parameter initialization.
Revised: In particular, if the test example is orthogonal to ˆ x (i.e., ˆ x (cid:62) x = 0 ), the prediction depends solely on the randomly initialized values in W 0 and therefore can be characterized by the distribution used for parameter initialization.

Original: However, when both layers are trained, the upper layer weights are also tuned to make the prediction ﬁt the training output.
Revised: However, when both layers are trained, the upper layer weights are also tuned to make the prediction ﬁt the training output.

Original: In particular, the learned weights in the upper layer depend on W 0 .
Revised: In particular, the learned weights in the upper layer depend on W 0 .

Original: Therefore, the randomness arguments shown above no longer apply even for test examples orthogonal to ˆ x .
Revised: Therefore, the randomness arguments shown above no longer apply even for test examples orthogonal to ˆ x .

Original: As the empirical results show, the behavior is indeed different.
Revised: As the empirical results show, the behavior is indeed different.

Original: E. N ONLINEAR MULTI - LAYER FULLY CONNECTED NETWORKS
Revised: E. N ONLINEAR MULTI - LAYER FULLY CONNECTED NETWORKS

Original: In this section, we consider the general case of multilayer fully connected networks (FCNs) with
Revised: In this section, we consider the general case of multilayer fully connected networks (FCNs) with

Original: ReLU activation functions.
Revised: ReLU activation functions.

Original: Figure 13 visualizes predictions from trained ReLU FCNs with 1, 3, or
Revised: Figure 13 visualizes predictions from trained ReLU FCNs with 1, 3, or

Original: [Figure]
Revised: [Figure]

Original: hidden layers.
Revised: hidden layers.

Original: The deepest network encodes the constant map with high conﬁdence; the shallowest network shows behavior similar to that of a one-layer linear net.
Revised: The deepest network encodes the constant map with high conﬁdence; the shallowest network shows behavior similar to that of a one-layer linear net.

Original: Quantitative evaluations of the behaviors are shown in Figure 14, computed in the same way as Figure 4 for CNNs (see Section 3.2).
Revised: Quantitative evaluations of the behaviors are shown in Figure 14, computed in the same way as Figure 4 for CNNs (see Section 3.2).

Original: The results for linear FCNs are also shown for comparison.
Revised: The results for linear FCNs are also shown for comparison.

Original: The linear and ReLU FCNs behave similarly when measuring the correlation to the identity function: neither of them performs well for test images that are nearly orthogonal to ˆ x .
Revised: The linear and ReLU FCNs behave similarly when measuring the correlation to the identity function: neither of them performs well for test images that are nearly orthogonal to ˆ x .

Original: For the correlation to the constant function, ReLU FCNs overﬁt sooner than linear FCNs when the depth increases.
Revised: For the correlation to the constant function, ReLU FCNs overﬁt sooner than linear FCNs when the depth increases.

Original: This is consistent with our previous visual inspections: for shallow models, the networks learn neither the constant nor the identity function, as the predictions on nearly orthogonal examples are random.
Revised: This is consistent with our previous visual inspections: for shallow models, the networks learn neither the constant nor the identity function, as the predictions on nearly orthogonal examples are random.

Original: F V ISUALIZATION OF THE INTERMEDIATE LAYER REPRESENTATIONS FOR CNN S
Revised: F V ISUALIZATION OF THE INTERMEDIATE LAYER REPRESENTATIONS FOR CNN S

Original: Unlike the FCNs, CNNs preserve the spatial relation between neurons in the hidden layers, so we can easily visualize the intermediate layers as images in comparison to the inputs and outputs, to gain more insights on how the networks are computing the functions layer-by-layer.
Revised: Unlike the FCNs, CNNs preserve the spatial relation between neurons in the hidden layers, so we can easily visualize the intermediate layers as images in comparison to the inputs and outputs, to gain more insights on how the networks are computing the functions layer-by-layer.

Original: In Figure 15, we visualize the intermediate layer representations on some test patterns for CNNs with different depths.
Revised: In Figure 15, we visualize the intermediate layer representations on some test patterns for CNNs with different depths.

Original: In particular, for each example, the outputs from a convolutional layer in an intermediate layer is a three dimensional tensor of shape (#channel, height, width).
Revised: In particular, for each example, the outputs from a convolutional layer in an intermediate layer is a three dimensional tensor of shape (#channel, height, width).

Original: To get a compact visualization for multiple channels in each layer, we compute ﬂatten the 3D tensor to a matrix of shape (#channel, height × width), compute SVD and visualize the top singular vector as a one-channel image.
Revised: To get a compact visualization for multiple channels in each layer, we compute ﬂatten the 3D tensor to a matrix of shape (#channel, height × width), compute SVD and visualize the top singular vector as a one-channel image.

Original: In the ﬁrst column, we visualize a 20-layer CNN at random initialization 3 .
Revised: In the ﬁrst column, we visualize a 20-layer CNN at random initialization 3 .

Original: As expected, the randomly initialized convolutional layers gradually smooth out the input 
Revised: As expected, the randomly initialized convolutional layers gradually smooth out the input images

Original: However, it manages to recover meaningful information in the higher layer activations that were (visually) lost in the random initialization.
Revised: However, it manages to recover meaningful information in the higher layer activations that were (visually) lost in the random initialization.

Original: On the other hand, in the last column, the network is so deep that it fails to make connection from the input to the output.
Revised: On the other hand, in the last column, the network is so deep that it fails to make connection from the input to the output.

Original: Instead, the network start from scratch and constructs the digit ‘7’ from empty and predict everything as ‘7’.
Revised: Instead, the network start from scratch and constructs the digit ‘7’ from empty and predict everything as ‘7’.

Original: However, note that around layer-8, we see the activations depict slightly more clear structures than the randomly initialized network.
Revised: However, note that around layer-8, we see the activations depict slightly more clear structures than the randomly initialized network.

Original: This suggests that some efforts have been made during the learning, as opposed to the case that the bottom layers not being learned due to complete gradient vanishing.
Revised: This suggests that some efforts have been made during the learning, as opposed to the case that the bottom layers not being learned due to complete gradient vanishing.

Original: Please refer to
Revised: Please refer to

Original: Appendix G for further details related to potential gradient vanishing problems.
Revised: Appendix G for further details related to potential gradient vanishing problems.

Original: Two alternative visualizations to the intermediate multi-channel representations are provided that show the channel that is maximally correlated with the input image, and a random channel (channel 0).
Revised: Two alternative visualizations to the intermediate multi-channel representations are provided that show the channel that is maximally correlated with the input image, and a random channel (channel 0).

Original: Figure 16, Figure 17 and Figure 18 illustrate a 7-layer CNN, a 14-layer CNN and a 20-layer
Revised: Figure 16, Figure 17 and Figure 18 illustrate a 7-layer CNN, a 14-layer CNN and a 20-layer

Original: CNN, respectively.
Revised: CNN, respectively.

Original: G M EASURING THE CHANGE IN WEIGHTS OF LAYERS POST TRAINING
Revised: G M EASURING THE CHANGE IN WEIGHTS OF LAYERS POST TRAINING

Original: In this section, we study the connection between the inductive bias of learning the constant function and the potential gradient vanishing problem.
Revised: In this section, we study the connection between the inductive bias of learning the constant function and the potential gradient vanishing problem.

Original: Instead of measuring the norm of gradient during training, we use a simple proxy that directly compute the distance of the weight tensor before and after training.
Revised: Instead of measuring the norm of gradient during training, we use a simple proxy that directly compute the distance of the weight tensor before and after training.

Original: In particular, for each weight tensor W 0 at initialization and W (cid:63) after training, we compute the relative (cid:96) 2 distance as
Revised: In particular, for each weight tensor W 0 at initialization and W (cid:63) after training, we compute the relative (cid:96) 2 distance as

Original: [Equation]
Revised: [Equation]

Original: The results for CNNs with various depths are plotted in Figure 19(a).
Revised: The results for CNNs with various depths are plotted in Figure 19(a).

Original: As a general pattern, we do see that as the network architecture gets deeper, the distances at lower layers do become smaller.
Revised: As a general pattern, we do see that as the network architecture gets deeper, the distances at lower layers do become smaller.

Original: But they are still non-zero, which is consistent with the visualization in Figure 15 showing that even
Revised: But they are still non-zero, which is consistent with the visualization in Figure 15 showing that even

Original: [Figure]
Revised: [Figure]

Original: for the 20-layer CNN, where the output layer ﬁts to the constant function, the lower layers does get enough updates to allow them to be visually distinguished from the random initialization.
Revised: for the 20-layer CNN, where the output layer ﬁts to the constant function, the lower layers does get enough updates to allow them to be visually distinguished from the random initialization.

Original: In Figure 19(b) and (c), we show the same plots for linear FCNs and FCNs with ReLU activation, respectively.
Revised: In Figure 19(b) and (c), we show the same plots for linear FCNs and FCNs with ReLU activation, respectively.

Original: We see that especially for ReLU FCN with 11 hidden layers, the distances for the weight tensors at the lower 5 layers are near zero.
Revised: We see that especially for ReLU FCN with 11 hidden layers, the distances for the weight tensors at the lower 5 layers are near zero.

Original: However, recall from Figure 13 in Section E.3, the ReLU FCNs start to bias towards the constant function with only three hidden layers, which are by no means suffering from vanishing gradients as the plots here demonstrate.
Revised: However, recall from Figure 13 in Section E.3, the ReLU FCNs start to bias towards the constant function with only three hidden layers, which are by no means suffering from vanishing gradients as the plots here demonstrate.

Original: H F ULL RESULTS ON ROBUSTNESS OF INDUCTIVE BIASES H. T ESTING ON DIFFERENT INPUT IMAGE SIZES
Revised: H F ULL RESULTS ON ROBUSTNESS OF INDUCTIVE BIASES H. T ESTING ON DIFFERENT INPUT IMAGE SIZES

Original: Figure 6 in Section 3.3 visualize the predictions of a 5-layer CNN on inputs of the extreme sizes of 7 × 7 and 112 × 112 .
Revised: Figure 6 in Section 3.3 visualize the predictions of a 5-layer CNN on inputs of the extreme sizes of 7 × 7 and 112 × 112 .

Original: Here in Figure 20 we provide more results on some intermediate image sizes.
Revised: Here in Figure 20 we provide more results on some intermediate image sizes.

Original: We also test a trained 20-layer CNN on various input sizes (Figure 7).
Revised: We also test a trained 20-layer CNN on various input sizes (Figure 7).

Original: It is interesting that the constant map also holds robustly across a wide range of input sizes.
Revised: It is interesting that the constant map also holds robustly across a wide range of input sizes.

Original: Especially from the prediction on large input images, we found hints on how the CNNs actually compute the constant function.
Revised: Especially from the prediction on large input images, we found hints on how the CNNs actually compute the constant function.

Original: It seems the artifacts from the outside boundary due to convolution padding are used as cues to “grow” a pattern inwards to generate the digit “7”.
Revised: It seems the artifacts from the outside boundary due to convolution padding are used as cues to “grow” a pattern inwards to generate the digit “7”.

Original: The visualizations of the intermediate layer representatiosn in Figure 21 is consistent with our hypothesis: the CNN take the last several layers to realize this construction.
Revised: The visualizations of the intermediate layer representatiosn in Figure 21 is consistent with our hypothesis: the CNN take the last several layers to realize this construction.

Original: This is very clever, because in CNNs, the same ﬁlter and bias is applied to all the spatial locations.
Revised: This is very clever, because in CNNs, the same ﬁlter and bias is applied to all the spatial locations.

Original: Without relying on the artifacts on the boundaries, it would be very challenging to get a sense of the spatial location in order to construct an image with a holistic structure (e.g. the digit “7” in our case).
Revised: Without relying on the artifacts on the boundaries, it would be very challenging to get a sense of the spatial location in order to construct an image with a holistic structure (e.g. the digit “7” in our case).

Original: H. T HE UPPER SUBNETWORKS
Revised: H. T HE UPPER SUBNETWORKS

Original: In the visualization of intermediate layers (Figure 15), the intermediate layers actually represent the “lower” subnetwork from the inputs.
Revised: In the visualization of intermediate layers (Figure 15), the intermediate layers actually represent the “lower” subnetwork from the inputs.

Original: Here we investigate the “upper” subnetwork.
Revised: Here we investigate the “upper” subnetwork.

Original: Thanks again to the spatial structure of CNNs, we can skip the lower layers and feed the test patterns directly to the intermediate layers and still get interpretable visualizations 4 .
Revised: Thanks again to the spatial structure of CNNs, we can skip the lower layers and feed the test patterns directly to the intermediate layers and still get interpretable visualizations 4 .

Original: Figure 22 shows the results for the top-one layer from CNNs with various depths.
Revised: Figure 22 shows the results for the top-one layer from CNNs with various depths.

Original: A clear distinction can be found at 15-layer CNN,
Revised: A clear distinction can be found at 15-layer CNN,

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: which according to Figure 3 is where the networks start to bias away from edge detector and towards the constant function.
Revised: which according to Figure 3 is where the networks start to bias away from edge detector and towards the constant function.

Original: The predictions from the ﬁnal two layers of each network are visualized in Figure 23.
Revised: The predictions from the ﬁnal two layers of each network are visualized in Figure 23.

Original: Figurefocuses on the 20-layer CNN that learns the constant map, and visualize the upper 3 layers, 6 layers and 10 layers, respectively.
Revised: Figurefocuses on the 20-layer CNN that learns the constant map, and visualize the upper 3 layers, 6 layers and 10 layers, respectively.

Original: In particular, the last visualization shows that the 20-layer CNN is already starting to construct the digit “7” from nowhere when using only the upper half of the model. 
Revised: In particular, the last visualization shows that the 20-layer CNN is already starting to construct the digit “7” from nowhere when using only the upper half of the model. [Figure]

Original: [Figure]
Revised: [Figure]

Revised: t

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: i 
Revised: i z

Original: [Figure]
Revised: [Figure]

Original: I F ULL RESULTS ON VARYING DIFFERENT FACTORS WHEN TRAINING CNN S
Revised: I F ULL RESULTS ON VARYING DIFFERENT FACTORS WHEN TRAINING CNN S

Original: The study on the effects of various hyperparameters on the inductive bias of trained CNNs is brieﬂy presented in Section 3.4.
Revised: The study on the effects of various hyperparameters on the inductive bias of trained CNNs is brieﬂy presented in Section 3.4.

Original: The full results are shown here.
Revised: The full results are shown here.

Original: Convolution ﬁlter sizes Figure 25 and Figure 26 illustrate the inductive bias with varying convolution ﬁlter size from 5 × 5 to 57 × 57 .
Revised: Convolution ﬁlter sizes Figure 25 and Figure 26 illustrate the inductive bias with varying convolution ﬁlter size from 5 × 5 to 57 × 57 .

Original: The visualization shows that the predictions become more and more blurry as the ﬁlter sizes grow.
Revised: The visualization shows that the predictions become more and more blurry as the ﬁlter sizes grow.

Original: The heatmaps, especially the correlation to the identity function, are not as helpful in this case as the correlation metric is not very good at distinguishing images with different levels of blurry.
Revised: The heatmaps, especially the correlation to the identity function, are not as helpful in this case as the correlation metric is not very good at distinguishing images with different levels of blurry.

Original: With extremely large ﬁlter sizes that cover the whole inputs, the CNNs start to bias towards the constant function.
Revised: With extremely large ﬁlter sizes that cover the whole inputs, the CNNs start to bias towards the constant function.

Original: Note our training inputs are of size 28 × 28 , so 29 × 29 ﬁlter size allows all the neurons to see no less than half of the spatial domain from the previous layer.
Revised: Note our training inputs are of size 28 × 28 , so 29 × 29 ﬁlter size allows all the neurons to see no less than half of the spatial domain from the previous layer.

Original: 57 × 57 ﬁlters centered at any location within the image will be able to see the whole previous layer.
Revised: 57 × 57 ﬁlters centered at any location within the image will be able to see the whole previous layer.

Original: On the other hand, the repeated application of the same convolution ﬁlter through out the spatial domain is still used (with very large boundary paddings in the inputs).
Revised: On the other hand, the repeated application of the same convolution ﬁlter through out the spatial domain is still used (with very large boundary paddings in the inputs).

Original: So the CNNs are not trivially doing the same computation as FCNs.
Revised: So the CNNs are not trivially doing the same computation as FCNs.

Original: Convolution channel depths Figure 10 in Section 3.4 shows that the learned identity function is severely corrupted when only 3 channels are used in the 5-layer CNNs.
Revised: Convolution channel depths Figure 10 in Section 3.4 shows that the learned identity function is severely corrupted when only 3 channels are used in the 5-layer CNNs.

Original: Figure 27 presents the correlation to the constant and the identity function when different numbers of convolution channels are used.
Revised: Figure 27 presents the correlation to the constant and the identity function when different numbers of convolution channels are used.

Original: The heatmap is consistent with the visualizations, showing that the 5-layer CNN fails to approximate the identity function when only three channels are used in each convolution layer.
Revised: The heatmap is consistent with the visualizations, showing that the 5-layer CNN fails to approximate the identity function when only three channels are used in each convolution layer.

Original: Furthermore, Figure 28 visualize the predictions of trained 3-channel CNNs with various depths.
Revised: Furthermore, Figure 28 visualize the predictions of trained 3-channel CNNs with various depths.

Original: The 3-channel CNNs beyond 8 layers fail to converge during training.
Revised: The 3-channel CNNs beyond 8 layers fail to converge during training.

Original: The 5-layer and the 7-layer
Revised: The 5-layer and the 7-layer

Original: CNNs implement functions biased towards edge-detecting or countour-ﬁnding.
Revised: CNNs implement functions biased towards edge-detecting or countour-ﬁnding.

Original: But the 6-layer and the 8-layer CNNs demonstrate very different biases.
Revised: But the 6-layer and the 8-layer CNNs demonstrate very different biases.

Original: The potential reason is that with only a few
Revised: The potential reason is that with only a few

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: channels, the random initialization does not have enough randomness to smooth out “unlucky” bad cases.
Revised: channels, the random initialization does not have enough randomness to smooth out “unlucky” bad cases.

Original: Therefore, the networks have higher chance to converge to various corner cases.
Revised: Therefore, the networks have higher chance to converge to various corner cases.

Original: Figureand Figure 30 compare the random initialization with the converged network for a 3-channel CNN and a 128-channel CNN.
Revised: Figureand Figure 30 compare the random initialization with the converged network for a 3-channel CNN and a 128-channel CNN.

Original: From the visualizations of the intermediate layers, the 128-channel CNN already behave more smoothly than the 3-channel CNN at initialization.
Revised: From the visualizations of the intermediate layers, the 128-channel CNN already behave more smoothly than the 3-channel CNN at initialization.

Original: Initialization schemes In transfer learning, it is well known that initializing with pre-trained network weights could affect the inductive bias of trained models.
Revised: Initialization schemes In transfer learning, it is well known that initializing with pre-trained network weights could affect the inductive bias of trained models.

Original: On the other hand, different random initialization schemes are mainly proposed to help with optimization by maintaining information ﬂow or norms of representations at different depths.
Revised: On the other hand, different random initialization schemes are mainly proposed to help with optimization by maintaining information ﬂow or norms of representations at different depths.

Original: It turns out that they also strongly affect the inductive bias of the trained models.
Revised: It turns out that they also strongly affect the inductive bias of the trained models.

Original: Figure 31 visualizes the different inductive biases.
Revised: Figure 31 visualizes the different inductive biases.

Original: Let f i , f o be the fan in and fan out of the layer being initialized.
Revised: Let f i , f o be the fan in and fan out of the layer being initialized.

Original: We tested the following commonly used initialization schemes: 1) default: N (0 , σ 2 = 1 (cid:14) ( f i f o )) ; 2) Xavier (a.k.a.
Revised: We tested the following commonly used initialization schemes: 1) default: N (0 , σ 2 = 1 (cid:14) ( f i f o )) ; 2) Xavier (a.k.a.

Original: Glorot) init (Glorot & Bengio, 2010): N (0 , σ 2 = 2 (cid:14) ( f i + f o )) ;
Revised: Glorot) init (Glorot & Bengio, 2010): N (0 , σ 2 = 2 (cid:14) ( f i + f o )) ;

Original: 3) Kaiming init (He et al., 2015): N (0 , σ 2 = 2 (cid:14) f i ) ; 4) Orthogonal init (Saxe et al., 2014).
Revised: 3) Kaiming init (He et al., 2015): N (0 , σ 2 = 2 (cid:14) f i ) ; 4) Orthogonal init (Saxe et al., 2014).

Original: Variations with uniform distributions instead of Gaussian distributions are also evaluated for Xavier and Kaiming inits.
Revised: Variations with uniform distributions instead of Gaussian distributions are also evaluated for Xavier and Kaiming inits.

Original: All initialization schemes bias toward the identity function for shallow networks.
Revised: All initialization schemes bias toward the identity function for shallow networks.

Original: But Kaiming init produces heavy artifacts on test predictions.
Revised: But Kaiming init produces heavy artifacts on test predictions.

Original: For the bias towards the constant function in deep networks, Xavier init behaves similarly to the default init scheme, though more layers are needed to learn a visually good identity function.
Revised: For the bias towards the constant function in deep networks, Xavier init behaves similarly to the default init scheme, though more layers are needed to learn a visually good identity function.

Original: On the other hand, the corresponding results from the Kaiming init is less interpretable.
Revised: On the other hand, the corresponding results from the Kaiming init is less interpretable.

Original: Optimizers First order stochastic optimizers are dorminately used in deep learning due to the huge model sizes and dataset sizes.
Revised: Optimizers First order stochastic optimizers are dorminately used in deep learning due to the huge model sizes and dataset sizes.

Original: To improve convergence speed, various adaptive methods are introduced (Duchi et al., 2011; Kingma & Ba, 2014; Graves, 2013).
Revised: To improve convergence speed, various adaptive methods are introduced (Duchi et al., 2011; Kingma & Ba, 2014; Graves, 2013).

Original: It is known that those methods lead to worse generalization performances in some applications (e.g. Wu et al.
Revised: It is known that those methods lead to worse generalization performances in some applications (e.g. Wu et al.

Original: But they are extremely popular in practice due to the superior convergence speed and easier hyper-parameter tuning than the vanilla SGD.
Revised: But they are extremely popular in practice due to the superior convergence speed and easier hyper-parameter tuning than the vanilla SGD.

Original: We compare several popular optimizers in our framework, and conﬁrm that different optimizers ﬁnd different global minimizers, and those minimizers show drastically
Revised: We compare several popular optimizers in our framework, and conﬁrm that different optimizers ﬁnd different global minimizers, and those minimizers show drastically

Original: D
Revised: D

Original: Xn
Revised: Xn

Original: Xu
Revised: Xu

Original: Kn
Revised: Kn

Original: Ku
Revised: Ku

Original: Or
Revised: Or

Original: [Figure]
Revised: [Figure]

Original: D
Revised: D

Original: Xn
Revised: Xn

Original: Xu
Revised: Xu

Original: Kn
Revised: Kn

Original: Ku
Revised: Ku

Original: Or
Revised: Or

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: D
Revised: D

Original: Xn
Revised: Xn

Original: Xu
Revised: Xu

Original: Kn
Revised: Kn

Original: Ku
Revised: Ku

Original: Or
Revised: Or

Original: [Figure]

Original: [Figure]

Original: r

Original: [Figure]

Original: s

Original: [Figure]

Original: a

Original: [Figure]
Revised: [Figure]

Original: a

Original: [Figure]
Revised: [Figure]

Original: a d a g r m s p a d a m a d a m x
Revised: a d a g r m s p a d a m a d a m x

Original: [Figure]
Revised: [Figure]

Original: different inductive biases on test examples, as shown in Figure 32.
Revised: different inductive biases on test examples, as shown in Figure 32.

Original: Some optimizer requires a smaller base learning rate to avoid parameter exploding during training.
Revised: Some optimizer requires a smaller base learning rate to avoid parameter exploding during training.

Original: In particular, we use base learning rate 0.001 for Adagrad and Adamax, 0.0001 for Adam and RMSprop.
Revised: In particular, we use base learning rate 0.001 for Adagrad and Adamax, 0.0001 for Adam and RMSprop.

Original: For comparison, we also include results from SGD with those corresponding base learning rates.
Revised: For comparison, we also include results from SGD with those corresponding base learning rates.

Original: J C ORRELATION VS MSE
Revised: J C ORRELATION VS MSE

Original: Figure 33, Figure 34 and Figure 35 can be compared to their corresponding ﬁgures in the main text.
Revised: Figure 33, Figure 34 and Figure 35 can be compared to their corresponding ﬁgures in the main text.

Original: The ﬁgures here are plotted with the MSE metric between the prediction and the groundtruth, while the ﬁgures in the main text uses the correlation metric.
Revised: The ﬁgures here are plotted with the MSE metric between the prediction and the groundtruth, while the ﬁgures in the main text uses the correlation metric.

Original: Each corresponding pair of plots are overall consistent.
Revised: Each corresponding pair of plots are overall consistent.

Original: But the correlation plots show the patterns more clearly and has a ﬁxed value range of [0, 1] that is easier to interpret.
Revised: But the correlation plots show the patterns more clearly and has a ﬁxed value range of [0, 1] that is easier to interpret.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: Input 1-layer 3-layer 5-layer 7-layer 9-layer 11-layer 12-layer 14-layer 15-layer 16-layer 18-layer 19-layer 20-layer mnist-n1-conv-rep (a) 2nd run
Revised: Input 1-layer 3-layer 5-layer 7-layer 9-layer 11-layer 12-layer 14-layer 15-layer 16-layer 18-layer 19-layer 20-layer mnist-n1-conv-rep (a) 2nd run

Original: Input 1-layer 3-layer 5-layer 7-layer 9-layer 11-layer 12-layer 14-layer 15-layer 16-layer 18-layer 19-layer 20-layer mnist-n1-conv-rep
Revised: Input 1-layer 3-layer 5-layer 7-layer 9-layer 11-layer 12-layer 14-layer 15-layer 16-layer 18-layer 19-layer 20-layer mnist-n1-conv-rep

Original: [Figure]
Revised: [Figure]

Original: Input

Original: [Figure]
Revised: [Figure]

Original: layer

Original: [Figure]
Revised: [Figure]

Original: layer

Original: [Figure]
Revised: [Figure]

Original: layer
Revised: layer

Original: [Figure]
Revised: [Figure]

Original: layer 14-layer

Original: [Figure]
Revised: [Figure]

Original: layer
Revised: 18-layer

Original: [Figure]
Revised: [Figure]

Original: layer 18-layer 19-layer 20-

Original: [Figure]
Revised: [Figure]

Original: mnist-n1-conv-rep
Revised: mnist-n1-conv-rep

Original: [Figure]
Revised: [Figure]

Original: Input1-layer
Revised: Input

Original: [Figure]
Revised: [Figure]

Original: -layer
Revised: -layer

Original: [Figure]
Revised: [Figure]

Original: -layer
Revised: -layer

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 11-layer 12-layer [Figure] [Figure] [Figure]
Revised: -layer

Original: [Figure]

Original: 18-layer

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: K R OBUSTNESS OF OBSERVATIONS TO RANDOM SEEDS
Revised: K R OBUSTNESS OF OBSERVATIONS TO RANDOM SEEDS

Original: We evaluate the robustness of our main observations to randomness from the experiments in this section.
Revised: We evaluate the robustness of our main observations to randomness from the experiments in this section.

Original: In Figure 36, two different runs of training and evaluation are compared side by side, and the results are very consistent.
Revised: In Figure 36, two different runs of training and evaluation are compared side by side, and the results are very consistent.

Original: In Figure 37 we further perturb the random seeds for data loading, so that different (single) training images are loaded for training.
Revised: In Figure 37 we further perturb the random seeds for data loading, so that different (single) training images are loaded for training.

Original: We can see that the main observations
Revised: We can see that the main observations

Original: [Figure]
Revised: [Figure]

Original: hold for both cases: CNNs learn the identity map, edge detector and the constant map as the depth increases.
Revised: hold for both cases: CNNs learn the identity map, edge detector and the constant map as the depth increases.

Original: L L EARNING WITH MULTIPLE EXAMPLES
Revised: L L EARNING WITH MULTIPLE EXAMPLES

Original: We focus on learning from a single example in this paper because the two extreme inductive biases of the identity map and the constant map can be precisely deﬁned and tested against.
Revised: We focus on learning from a single example in this paper because the two extreme inductive biases of the identity map and the constant map can be precisely deﬁned and tested against.

Original: When training with multiple examples, the constant map is no longer a viable solution to the optimization problem.
Revised: When training with multiple examples, the constant map is no longer a viable solution to the optimization problem.

Original: Nevertheless, we can still qualitatively evaluate the inductive bias via visualization of predictions.
Revised: Nevertheless, we can still qualitatively evaluate the inductive bias via visualization of predictions.

Original: Figure 38 shows the results on various networks trained with two examples.
Revised: Figure 38 shows the results on various networks trained with two examples.

Original: In particular, for networks that are known to overﬁt to the constant map when trained with one example (e.g. fully connected networks with 9 hidden layers, and 20-layer convolutional networks), similar overﬁtting behaviors are observed.
Revised: In particular, for networks that are known to overﬁt to the constant map when trained with one example (e.g. fully connected networks with 9 hidden layers, and 20-layer convolutional networks), similar overﬁtting behaviors are observed.

Original: In this case, a proper deﬁnition of a constant map no longer holds, as the network perfectly reconstruct each individual digit from the training set.
Revised: In this case, a proper deﬁnition of a constant map no longer holds, as the network perfectly reconstruct each individual digit from the training set.

Original: On the test set, a notion of memorization can be recognized as always predicting a mixture of the training samples.
Revised: On the test set, a notion of memorization can be recognized as always predicting a mixture of the training samples.

Original: Note sometimes the prediction is biased more towards one of the training samples depending on the input patterns.
Revised: Note sometimes the prediction is biased more towards one of the training samples depending on the input patterns.

Original: Figure 39 shows the results with three examples, with similar observations.
Revised: Figure 39 shows the results with three examples, with similar observations.

Original: Figure 40 shows the situation when training with the full MNIST training set (60k examples).
Revised: Figure 40 shows the situation when training with the full MNIST training set (60k examples).

Original: In this case, even the deepest convolutional networks we tried successfully learn the identity function.
Revised: In this case, even the deepest convolutional networks we tried successfully learn the identity function.

Original: Fully connected networks learn the identity function on the manifold of digit inputs, but still cannot reconstruct meaningful results on other test patterns.
Revised: Fully connected networks learn the identity function on the manifold of digit inputs, but still cannot reconstruct meaningful results on other test patterns.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: M P ARAMETER COUNTS FOR NEURAL NETWORKS USED IN THIS PAPER
Revised: M P ARAMETER COUNTS FOR NEURAL NETWORKS USED IN THIS PAPER

Original: For convenience of cross comparing the results of different architectures with similar number of parameters, we list in Table 1 the architectures used in this paper and their associated number of parameters
Revised: For convenience of cross comparing the results of different architectures with similar number of parameters, we list in Table 1 the architectures used in this paper and their associated number of parameters

Original: . N R ESIDUAL N ETWORKS
Revised: . N R ESIDUAL N ETWORKS

Original: In this section, we evaluate training FCNs with residual connections.
Revised: In this section, we evaluate training FCNs with residual connections.

Original: In particular, an identity skip connection is added for every two fully connected layers.
Revised: In particular, an identity skip connection is added for every two fully connected layers.

Original: In other words, the networks are built with two-layer blocks that computes x (cid:55)→ x + ReLU ( W 2 ReLU ( W 1 x )) , except that no ReLU is applied at the output layer.
Revised: In other words, the networks are built with two-layer blocks that computes x (cid:55)→ x + ReLU ( W 2 ReLU ( W 1 x )) , except that no ReLU is applied at the output layer.

Original: Adding skip connection after ReLU ensures the input gets passed directly to the output layer.
Revised: Adding skip connection after ReLU ensures the input gets passed directly to the output layer.

Original: Because the residual structure requires the same input-output shape for every block, we use 784 hidden dimensions.
Revised: Because the residual structure requires the same input-output shape for every block, we use 784 hidden dimensions.

Original: Comparing Figure 41 with the vanilla FCNs in Figure 2, the identity skip connection strongly biased the FCNs towards learning the identity map.
Revised: Comparing Figure 41 with the vanilla FCNs in Figure 2, the identity skip connection strongly biased the FCNs towards learning the identity map.

Original: On the other hand, we can still observe that the prediction noises become stronger with larger number of hidden layers, suggesting that learning the identity function is non-trivial even with the explicit identity skip connections built into the architectures.
Revised: On the other hand, we can still observe that the prediction noises become stronger with larger number of hidden layers, suggesting that learning the identity function is non-trivial even with the explicit identity skip connections built into the architectures.

Original: O E XPERIMENTS ON CIFAR-10 I MAGES
Revised: O E XPERIMENTS ON CIFAR-10 I MAGES

Original: In this section, we show experiments on some colored images from the CIFAR-10 dataset ( 32 ×
Revised: In this section, we show experiments on some colored images from the CIFAR-10 dataset ( 32 ×

Original: RGB images).
Revised: RGB images).

Original: Figure 42 and Figure 43 show the results from two different randomly sampled training images, respectively.
Revised: Figure 42 and Figure 43 show the results from two different randomly sampled training images, respectively.

Original: The network architectures used here are nearly identical to the ones used to train on MNIST digits in the main text of the paper: the CNNs are with 128 channels and 5 ×
Revised: The network architectures used here are nearly identical to the ones used to train on MNIST digits in the main text of the paper: the CNNs are with 128 channels and 5 ×

Original: [Figure]
Revised: [Figure]

Original: ﬁlters.
Revised: ﬁlters.

Original: The FCNs are slightly modiﬁed to have larger hidden dimensions 3072 =
Revised: The FCNs are slightly modiﬁed to have larger hidden dimensions 3072 =

Original: 32 × 32 × 3 , to avoid explicitly enforcing bottlenecks in the hidden representations.
Revised: 32 × 32 × 3 , to avoid explicitly enforcing bottlenecks in the hidden representations.

Original: The training loss and optimization algorithms are the same as in the MNIST case.
Revised: The training loss and optimization algorithms are the same as in the MNIST case.

Original: The main observations are consistent the results on the MNIST digits.
Revised: The main observations are consistent the results on the MNIST digits.

Original: In particular, shallow FCNs produce random noisy predictions on unseen evaluation inputs, but deep FCNs bias towards memorization and hallucinate the training image on all test outputs.
Revised: In particular, shallow FCNs produce random noisy predictions on unseen evaluation inputs, but deep FCNs bias towards memorization and hallucinate the training image on all test outputs.

Original: For CNNs, shallow networks are capable of learning the identity function, but deep networks learn the constant function instead.
Revised: For CNNs, shallow networks are capable of learning the identity function, but deep networks learn the constant function instead.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]