Identity Crisis: Memorization and Generalization Under Extreme Overparameterization
Abstract
We study the interplay between memorization and generalization of overparametrized networks in the extreme case of a single training example.
The learning task is to predict an output which is as similar as possible to the input.
We examine both fully-connected and convolutional networks that are initialized randomly and then trained to minimize the reconstruction error.
The trained networks take one of the two forms: the constant function (“memorization”) and the identity function (“generalization”).
We show that different architectures exhibit vastly different inductive bias towards memorization and generalization.
Introduction
The empirical success of deep networks is often attributed to large data sets for training.
However, sample size does not provide a comprehensive rationale since complex models often outperform simple models on a given data set, even when the number of free parameters of each far exceeds the number of training examples.
What form of inductive bias could achieve state-of-the-art performance from highly overparameterized models?
Numerous theoretical and empirical studies of inductive bias in deep learning have been conducted in recent years [12, 16, 5, 23, 20, 24, 3, 31] but these postmortem analyses do not identify the root source of the bias.
One cult belief among researchers is that gradient-based optimization methods provide an implicit bias toward simple solutions.
However, when a network is sufficiently large, gradient methods provably fit the training set [2, 9, 10, 32].
These results do not distinguish a model trained on a data distribution with strong statistical regularities from one trained on randomly shuffled labels.
Although the former model might achieve good generalization, the latter can only memorize the training labels.
Consequently, these analyses do not shed much light on the question of inductive bias.
Another line of research characterizes sufficient conditions on the data distribution that guarantee generalization from a trained network [8, 19].
This direction, while very promising, has thus far identified only structures that can be solved by linear or nearest neighbor classifiers operating in the original input space.
The fact that in many applications deep neural networks significantly outperform these simpler models reveals a gap in our understanding.
In this article, we conduct a novel empirical exploration in a highly restrictive setting that admits visualization and quantification of inductive bias, allowing us to compare variations in architecture, network depth, optimization procedure, initialization scheme, and hyperparameters.
The particular task we investigate is learning an identity mapping in a regression setting.
The identity mapping is interesting for several reasons.
First, it imposes a structural regularity between the input and output, the type of regularity that could in principle lead to systematic generalization.
Second, it requires that every input feature is transmitted to the output and thus provides a sensitive indicator of whether a model succeeds in passing activations (and gradients) between inputs and outputs.
Third, conditional image generation is a popular task in the literature [e.g., 21, 18]; an identity mapping is the simplest form of such a generative process.
Fourth, and perhaps most importantly, it affords analysis and visualization of model behavior.
Consider networks trained on the identity task with 60k MNIST digits.
Although only digit images are presented during the training, one might expect the strong regularity of the task to lead to good generalization to images other [Figure] [Figure] than digits.
Figure 1 compares a 20-layer ConvNet, a 10-layer fully connected net with ReLU activation, and a 1-layer fully-connected net.
All nets perform well on the training set (first three columns) and transfer well to novel digits and digit blends (columns 4–6), but outside of the manifold of hand-printed digits, only the ConvNet discovers a reasonably good approximation to the identity function.
Figure 1 reflects architecture-specific inductive bias that persists even with 60k training examples.
Despite this persistence, a model’s intrinsic bias is more likely to be revealed with a smaller training set.
In this article, we push this argument to the limit by studying learning with a single training example.
Our initial intuition was that a single example would be uninteresting because models would simply induce a constant output (e.g., via biases on output units).
Further, it seemed inconceivable that biases would be sufficiently strong to learn a mapping close to the identity.
Unexpectedly, our experiments show that model behavior is subtle and architecture dependent.
In a broad set of experiments, we highlight model features—including depth, initialization, and hyperparameters—that determine where a model lands on the continuum between memorization (i.e., learning a constant function) and generalization (i.e., learning the identity function).
The simplicity of the training scenario permits this quantitative characterization of inductive bias.
Learning the identity function under extreme overparameterization
The simplest neural network is a one layer fully connected linear model, f W ( x ) =
Wx , where x ∈ R d and W ∈ R d × d .
The optimization problem is convex and well understood.
Due to overparameterization, no unique solution exists.
However, given randomly initialized weights W 0 , gradient descent converges to a unique global minimizer:
[Equation]
(see Appendix D).
Consequently, the prediction of a model trained on ˆ x is fully determined: the test example x is decomposed into the component in the direction of ˆ x and its orthogonal component.
The component parallel to ˆ x will make the prediction look like ˆ x , whereas the orthogonal component entirely depends on the random initialization.
In this case, the learning algorithm has a strong inductive bias that yields a unique solution given W 0 despite the overparameterization.
However, this inductive bias does not magically lead to model generalization: the trained model fails to learn the identity function.
Although it predicts well when the input has a strong correlation with training example ˆ x , otherwise predictions are random.
In particular, when the test example x is orthogonal to ˆ x , the prediction is completely random.
The second row of Figure 2 shows an example of predictions from such a model. i n p u t [Figure] i n p u t
[Figure]
The deviations from the identity function are quite different in multilayer fully connected networks.
A multilayer linear network with no bottleneck in the hidden space has the same representational power as a 1-layer linear network, but its learning dynamics are nonconvex and that alters the inductive biases.
Figure 2 shows that a 6-layer linear network behaves more like a 6-layer ReLU network than like the 1-layer convex case, with a strong bias towards a constant function that maps every input to the single training image.
See Appendix E for more details.
The situation is very different for ConvNets (Figure 3a).
Compared to general fully connected FCNs, ConvNets have strong structural constraints that limit the receptive field of each neuron to a spatially local neighborhood and the same weights are re-used across the spatial map.
The two constraints indeed match the structure of the identity target function.
(Appendix C.3 explores other cases of using ConvNets to construct the identity function.)
The Figure shows that, except for some artifacts on the boundaries, ConvNets with a depth of up-to-5 hidden layers learn a good approximation to the identity function.
As the depth increases, the ConvNets bias towards the constant function.
Visualization of intermediate representations and quantitative evaluations can be found in Appendix F.
Robustness of inductive biases
The bias towards either constant or identity map is not explicitly encoded in the loss function, but rather is implicit in the network architecture, the learning algorithm, and other hyperparameters.
To assess the robustness of the inductive bias, we systematically manipulate critical factors in training and testing.
We focus on ConvNets, due to their proven performance and the interpretability of their hidden layers.
Figure 3b shows predictions of a 20-layer
ConvNet that has learned the constant function on a 28 × 28 image and is tested on a set of inputs scaled to varying resolutions (indicated to the left of each row).
The learned constant map holds up for input dimensions slightly smaller [Figure] [Figure] than the trained image.
On the other hand, as the input dimension increases, it is interesting to see smooth changes which reveal the network’s understanding of the “7”.
Because of the fully convolutional network structure, output units do not have individual biases at each spatial location.
Consequently, the network must learn to synthesize the “7” through multiple layers of transformations, a nontrivial generative task.
The Figure suggests that the network starts from the outside, where the artifacts due to the boundary padding in convolution provide spatial cues.
The visualization of the intermediate layers in Figure 19 of Appendix H.1 confirms this hypothesis.
See Appendix H for more details and other analysis.
We also explored the consequence of training on images of varying dimensions.
Figure 4a shows the correlation of the model output to the constant and the identity functions for models trained on 7 × 7, 14 × 14, and 28 × 28 images.
All three show more bias toward the constant function and less bias toward the identity function with increased network depth.
However, smaller input grids accentuate the effect, likely because the training image has fewer pixels to constrain the mapping.
Figure 4b explores the effect of varying the number of channels.
Two channels are sufficient to encode the identity function for grayscale inputs (Appendix C.3).
However, the inductive bias of a trained model depends on the channel count.
The aggressively over-parameterized network with 1024 channels ( ∼ 25M parameters per middle-layer convolution module) does not seem to suffer from overfitting, whereas the network with three channels misses the center content of many predicted images.
Underfitting is not an issue for the small network because it reconstructs the training image (first column) correctly.
Our results could be explained by assuming that a model’s behavior is very sensitive to the initialization of the channels.
With fewer channels, it is far more likely that all channels are initialized poorly for conveying information (see Appendix H.3).
Studies of different training initialization schemes indeed confirm that random initial conditions can have a big impact on the inductive bias (see Appendix H).
Conclusions
We presented empirical studies of the extreme case of overparameterization when learning from a single example.
We investigated the interplay between memorization and generalization in deep neural networks.
By restricting the learning task to the identity function, we sidestepped issues such as the underlying optimal Bayes error of the problem and the approximation error of the hypothesis classes.
This choice also facilitated rich visualization and intuitive interpretation of the trained models.
Under this setup, we investigate gradient-based learning procedures with explicit memorization-generalization characterization.
Our results indicate that different architectures exhibit vastly different inductive bias towards memorization and generalization.
Acknowledgments We would like to thank Kunal Talwar, Hanie Sedghi, and Rong Ge for helpful discussions.
A Related work
The consequences of overparameterized models in deep learning have been extensively studied in recently years, on the optimization landscape and convergence of SGD [2, 9, 10, 6, 32, 25], as well as the generalization guarantees under stronger structural assumptions of the data [19, 8, 1].
Another line of related work is the study of the implicit regularization effects of SGD on training overparameterized models [22, 30, 28, 27].
The traits of memorization in learning are also explicitly studied from various perspectives such as prioritizing learning of simple patterns [4] or perfect interpolation of the training set [7].
More recently, coincidently with the writing of this paper, Radhakrishnan et al.
[26] reported on the effects of the downsampling operator in convolutional auto-encoders on image memorization.
Their empirical framework is similar to ours, fitting ConvNets to the autoregression problem with few training examples.
We focus on investigating the general inductive bias in the extreme overparameterization case, and study a broader range of network types without enforcing a bottleneck in the architectures.
B Experiment details and hyper-parameters
We specify the experiment setups and the hyper-parameters here.
Unless otherwise specified in each study (e.g. when we explicitly vary the number of convolution channels), all the hyper-parameters are set according to the default values here.
The main study is done with the MNIST dataset.
It consists of grayscale images of hand written digits of size 28 × 28.
For training, we randomly sample one digit from the training set (a digit ‘7’) with a fixed random seed.
For testing, we use random images from the test set of MNIST and Fashion-MNIST, as well as algorithmically generated structured patterns and random images.
The training and test images are all normalized by mapping the pixel values in { 0 , 1 , . . .
, 255 } to [ 0 , 1 ] , and then standardize with the mean 0 .
1307 and standard deviation 0 .
3081 originally calculated on the (full) MNIST training set.
The models are trained by minimizing the mean squared error (MSE) loss with a vanilla SGD (base learning rate 0.
and momentum 0.9).
The learning rate is scheduled as stagewise constant that decays with a factor of 0.2 at the 30%, 60% and 80% of the total training steps (2,000,000).
No weight decay is applied during training.
For neural network architectures, the rectified linear unit (ReLU) activation is used for both fully connected networks (FCNs) and convolutional networks (ConvNets).
The input and output dimensions are decided by the data.
The hidden dimensions for the FCNs are 2,048 by default.
The ConvNets use 5 × 5 kernels with stride 1 and padding 2, so that the geometry does not change after each convolution layer.
C Representation of the identity function using deep networks
In this section, we provide explicit constructions on how common types of neural networks can represent the identity function.
Those constructions are only proof for that the models in our study have the capacity to represent the target function.
There are many different ways to construct the identity map for each network architecture, but we try to provide the most straightforward and explicit constructions.
However, during our experiments, even when the
SGD learns (approximately) the identity function, there is no evidence suggesting that it is encoding the functions in similar ways as described here.
We put some mild constraints (e.g. no “bottleneck” in the hidden dimensions) to allow more straightforward realization of the identity function, but this by no means asserts that networks violating those constraints cannot encode the identity function.
C. Linear models
For a one-layer linear network f ( x ) =
Wx , where W ∈ R d × d , setting W to the identity matrix will realize the identity function.
For a multi-layer linear network f ( x ) =
( (cid:206) (cid:96) W (cid:96) )
x , we need to require that all the hidden dimensions are not smaller than the input dimension.
In this case, a simple concrete construction is to set each W (cid:96) to an identity matrix.
C. Multi-layer ReLU networks
The ReLU activation function σ (·) = max ( 0 , ·) discards all the negative values.
There are many ways one can encode the negative values and recover it after ReLU.
We provide a simple approach that uses hidden dimensions twice the input dimension.
Consider a ReLU network with one hidden layer f ( x )
=
W 2 σ ( W 1 x ) , where W 2 ∈ R d × 2 d , W 1 ∈ R 2 d × d .
The idea is to store the positive and negative part of x separately, and then re-construct.
This can be achieved by setting
[Equation]
where I d is the d -dimensional identity matrix.
For the case of more than two layers, we can use the bottom layer to split the positive and negative part, and the top layer to merge them back.
All the intermediate layers can be set to 2 d -dimensional identity matrix.
Since the bottom layer encode all the responsives in non-negative values, the ReLU in the middle layers will pass through.
C. Convolutional networks
In particular, we consider 2D convolutional networks for data with the structure of multi-channel images.
A minibatch of data is usually formatted as a four-dimensional tensor of the shape B × C × H × W , where B is the batch size, C the number of channels (e.g. RGB or feature channels for intermediate layer representations), H and W are image height and width, respectively.
A convolutional layer (ignoring the bias term) is parameterized with another four-dimensional tensor of the shape ¯ C × C × K H
× K W , where ¯ C is the number of output feature channels, K H and
K W are convolutional kernel height and width, respectively.
The convolutional kernel is applied at local K H
× K W patches of the input tensor, with optional padding and striding.
For one convolution layer to represent the identity function, we can use only the center slice of the kernel tensor and set all the other values to zero.
Note it is very rare to use even numbers as kernel size, in which case the “center” of the kernel tensor is not well defined.
When the kernel size is odd, we can set
[Equation]
otherwise
By using only the center of the kernel, we essentially simulate a 1 × 1 convolution, and encode a local identity function for each (multi-channel) pixel.
For multi-layer convolutional networks with ReLU activation functions, the same idea as in multi-layer fully-connected networks can be applied.
Specifically, we ask for twice as many channels as the input channels for the hidden layers.
At the bottom layer, separately the positive and negative part of the inputs, and reconstruct them at the top layer.
D Closed form solution for single-layer overparameterized network
In (1), a closed form solution is provided for the global minimizer of a one-layer neural network trained on a single example.
The derivation of the solution is presented here.
Let ˆ x be the training example, the gradient of the empirical risk (the mean squared error on the single training example ˆ x ) is
[Equation]
[Equation]
Gradient descent with step sizes η t and initialization weights W 0 update weights as
[Equation]
where u T ∈ R
d is a vector decided via the accumulation in the optimization trajectory.
Because of the form of the gradient, it is easy to see the solution found by gradient descent will always have such parameterization structure.
[Figure]
Moreover, under this parameterization, a unique minimizer exists that solves the equation
[Equation]
via
[Equation]
Therefore, the global minimizer can be written as in (1), copied here for convenience:
[Equation]
For the one-layer network case, the optimization problem is convex.
Under standard conditions in convex optimization, gradient descent will converge to the global minimizer shown above.
The calculation can be easily extended to arbitrary target function other than the identity, as well as the case with multiple training examples.
It will result in a decomposition of the test example into the subspace spanned by the training samples, and the orthogonal subspace.
In particular, when the training examples have full rank — spanning the whole input space, the model will correctly learn the identity function.
E Full results of fully connected multi-layer networks E. Fully connected linear networks
Figure 5 shows the results on multi-layer linear networks with various number of hidden layers and hidden units.
The model with only one hidden layer resembles the convex case (see Figure 2), but as the depth increases, the model tends toward a constant function that maps every input to the single training image.
The depth of the architecture has a stronger effect on the inductive bias than the width.
For example, the network with one hidden layer of dimension 2048 has 3.2M parameters, more than the 2.5M parameters of the network with three hidden layers of dimension 784.
But the latter behaves less like the convex case.
E. Two-layer fully connected ReLU networks
Li and Liang [19] offer a theoretical characterization of learning in a two-layer ReLU neural network.
They show that when the data consist of well separated clusters (i.e., the cluster diameters are much smaller than the distances between each cluster pair), training an overparameterized two-layer ReLU network will generalize well.
To simplify
[Figure] [Figure]
We study the problem of learning a two-layer ReLU network under our identity-mapping task.
Figure 6 compares the cases of learning the bottom layer only and learning both layers (left and right panels, respectively).
The two cases demonstrate different inductive biases for predictions on unseen test images.
When only the first layer is trained, the tendency is toward speckled noise, but when both layers are trained, the tendency is toward a constant output (the image used for training).
Our observation does not contradict the theoretical results of Li and Liang [19], which assume a well separated and clustered data distribution.
For bottom-layer only training, the analysis is similar to that for one-layer networks.
Let us denote
[Equation]
where W ∈ R m × d is the learnable weight matrix, and α ∈ R m is randomly initialized and fixed.
Although the trained weights no longer have a closed-form solution, the solution found by gradient descent is always parameterized as
[Equation]
where ˆ x is the training example, and u T ∈ R m summarizes the efforts of gradient descent up to time T .
In particular, the gradient of the empirical risk ˆ R with respect to each row W : r of the learnable weight is
[Equation]
[Equation]
Putting it together, the full gradient is
[Equation]
Since the gradient lives in the span of the training example ˆ x , the solution found by gradient descent is always parameterized as (5).
The same arguments applies to multi-layer neural networks.
The prediction on any test example that is orthogonal to ˆ x will depend only on randomly initialized W 0 and upper layer weights.
When only the bottom layer is trained, the upper layer weights will also be independent from the data, therefore the prediction is completely random.
However, when all the layers are jointly trained, the arguments no longer apply.
The empirical results presented later in the paper that multi-layer networks bias towards the constant function verify this.
Consequently, if the test example is orthogonal to ˆ x (i.e., ˆ x (cid:62) x = 0), the prediction depends solely on the randomly initialized values in W 0 and therefore can be characterized by the distribution used for parameter initialization.
However, when both layers are trained, the upper layer weights are also tuned to make the prediction fit the training output.
In particular, the learned weights in the upper layer depend on W 0 .
Therefore, the randomness arguments shown above no longer apply even for test examples orthogonal to ˆ x .
As the empirical results show, the behavior is indeed different.
[Figure] [Figure]
for linear FCNs.
E. Nonlinear multi-layer fully connected networks
In this section, we consider the general case of multilayer fully connected networks (FCNs) with ReLU activation functions.
Figure 7 visualizes predictions from trained ReLU FCNs with 1, 3, or 9 hidden layers.
The deepest network encodes the constant map with high confidence; the shallowest network shows behavior similar to that of a linear net.
For a quantitative evaluation, we measure the performance via correlation 1 to the two reference functions: the identity function, and the constant function that maps every input to the training point ˆ x .
To evaluate the predictions on test images with varying similarity to the training image, we generate the test images according to the following procedure: for a given ρ ∈ [ 0 , 1 ] , generate a set of test images by sampling random pixels; then add to each image a component parallel to ˆ x to force the correlation with the training image to be ρ ; finally renormalize the test images to match the norm of the training image.
So for ρ = 0, the test images are orthogonal to ˆ x , while for ρ = 1, the test images equal ˆ x .
The results are shown in Figure 8.
The results for linear FCNs are also shown for comparison.
The linear and ReLU FCNs behave similarly when measuring the correlation to the identity function: neither of them performs well for test images that are nearly orthogonal to ˆ x .
For the correlation to the constant function, ReLU FCNs overfit sooner than linear FCNs when the depth increases.
This is consistent with our previous visual inspections:
[Figure]
rho: train-test correlation
C o n v
N e t
D e p t h
[Figure]
rho: train
[Figure]
test
[Figure]
correlation to identity function
[Figure] [Figure]
[Figure]
[Figure] [Figure]
[Figure]
F Detailed analysis for ConvNets
Quantitative evaluation for ConvNets with different depth are shown in Figure 9, computed in the same way as Figure 8.
The results are consistent with the visualizations: shallow ConvNets are able to learn the identity function from only one training example; very deep ConvNets bias towards the constant function; the ConvNets with intermediate depth correlate well with neither the identity nor the constant function.
However, unlike FCNs that produce white-noise like predictions, from the visualization we see that ConvNets behave like edge detectors.
To evaluate how much information is lost in the intermediate layer, we use the following simple criterion to measure the layerwise representations.
Take the MNIST dataset, for each layer in the network (trained on a single example), collect the representations obtained by feeding each item in the dataset through the network up to that layer, then do a simple similarity based global averaging classification and measure the error rate.
Specifically, the prediction for each test example is the argmax of the mean vector of the (one-hot) training labels, weighted by the correlation between the test example and eeach example from the MNIST training set, computed by the representation from the layer we want to inspect.
This metric does not quantify how much information is preserved as the representation propagate through layers in the information theoretical sense, as the information could still be present but encoded in complicated ways that
[Figure]
makes it hard for simple averaging-based classifier to pick out the signals.
But it provides a simple metric for our case: using the raw representation as the baseline, if a layer represents the identity function, then the representation at that layer will have similar error rate to the raw representation; on the other hand, if the a layer collapses to the constant function, then the corresponding representation will have error rate close to random.
The results are plotted in Figure 10.
The error rate curve for a randomly initialized 20-layer ConvNet is also shown as reference: at random initialization, the smoothing effect makes the representations beyond around layer 5-6 almost useless for our simple classifier.
After training, as the error rates for the output layers decrease, the curves generally form “concave” patterns.
The trained networks try to recover the smoothed out intermediate layer representations and make connections between the inputs and the outputs.
But if the gap is too big to push all the necessary information through, the network will try to infer the input-output relation using partial information, resulting in models that behave like edge detectors.
Finally, for the case of 20 layers, the curve shows that the bottom few layers do get small improvements in error rate, but the big gap between inputs and outputs drives the network to learn the constant function instead.
It looks like the vanishing gradient problem.
However, from Figure 1 we note that given enough training examples, a 20-layer ConvNet can still learn the identity map.
More studies on potential vanishing gradient issue can be found in Appendix G.
Because ConvNets preserves spatial structures, we can also qualitatively evaluation the information loss in the intermediate layers by directly visualizing them.
We found the behaviors consistent with the observations made here.
In particular, in Figure 11, we visualize the intermediate layer representations on some test patterns for ConvNets with different depth.
In particular, for each example, the outputs from a convolutional layer in an intermediate layer is a three dimensional tensor of shape (#channel, height, width).
To get a compact visualization for multiple channels in each layer, we compute SVD and visualize the top singular vector as a one-channel image.
In the first column, we visualize a 20-layer ConvNet at random initialization 2 .
As expected, the randomly initialized convolutional layers gradually smooth out the input images.
The shape of the input images are (visually) wiped out after around 8 layers of (random) convolution.
On the right of the figure, we show several trained ConvNets with increasing depths.
For a 7-layer ConvNet, the holistic structure of inputs are still visible all the way to the top at random initialization.
After training, the network approximately renders an identity function at the output, and the intermediate activations also become less blurry.
Next we show a 14-layer ConvNet, which fails to learn the identity function.
However, it manages to recover meaningful information in the higher layer activations that were (visually) lost in the random initialization.
On the other hand, in the last column, the network is so deep that it fails to make connection from the input to the output.
Instead, the network start from scratch and constructs the digit ‘7’ from empty and predict everything as ‘7’.
However, note that around layer-8, we see the activations depict slightly more clear structures than the randomly initialized network.
This suggests that some efforts have been made during the learning, as opposed to the case that the bottom layers not being learned due to complete gradient vanishing.
Two alternative visualizations to the intermediate multi-channel representations are provided that show the channel that is maximally correlated with the input image, and a random channel (channel 0).
Figure 12, Figure 13 and
Figure 14 illustrate a 7-layer ConvNet, a 14-layer ConvNet and a 20-layer ConvNet, respectively.
G Measuring the change in weights of layers post training
In this section, we study the connection between the inductive bias of learning the constant function and the potential gradient vanishing problem.
Instead of measuring the norm of gradient during training, we use a simple proxy that directly compute the distance of the weight tensor before and after training.
In particular, for each weight tensor W and initialization and W (cid:63) after training, we compute the relative (cid:96) 2 distance as
[Equation]
The results for ConvNets with various depths are plotted in Figure 15.
As a general pattern, we do see that as the network architecture gets deeper, the distances at lower layers do become smaller.
But they are still non-zero, which is consistent with the visualization in Figure 11 showing that even for the 20-layer ConvNet, where the output layer fits to the constant function, the lower layers does get enough updates to allow them to be visually distinguished from the random initialization.
In Figure 16 and Figure 17, we show the same plots for linear FCNs and FCNs with ReLU activation, respectively.
We see that especially for ReLU FCN with 11 hidden layers, the distances for the weight tensors at the lower 5 layers are near zero.
However, recall from Figure 7 in Section E.3, the ReLU FCNs start to bias towards the constant function with only three hidden layers, which are by no means suffering from vanishing gradients as the plots here demonstrate.
H Full results on robustness of inductive biases H. Testing on different input image sizes
Due to the perserved spatial structure in ConvNets, we can apply a trained network to a variety of different input sizes.
Figure 18 visualizes the predictions of a trained 5-layer ConvNets on images of sizes from 7 × 7 to 112 × 112.
We found that the learned identity map generally holds up against larger (than the training) input sizes.
However, on small inputs, the predictions no longer match well with the inputs.
Note ConvNets are in principle capable of encoding the identity function for arbitrary inputs and filter sizes, for example, via the construction in Appendix C.3.
In Section 3 we also test a trained 20-layer ConvNet on various input sizes (Figure 3b).
It is interesting that the constant map also holds robustly across a wide range of input sizes.
Especially from the prediction on large input images, we found hints on how the ConvNets actually compute the constant function.
It seems the artifacts from
layer index 0.
weight                      layer index 0.
bias                  the outside boundary due to convolution padding are used as cues to “grow” a pattern inwards to generate the digit “7”.
The visualizations of the intermediate layer representatiosn in Figure 19 is consistent with our hypothesis:
the
ConvNet take the last several layers to realize this construction.
This is very clever, because in ConvNets, the same filter and bias is applied to all the spatial locations.
Without relying on the artifacts on the boundaries, it would be very challenging to get a sense of the spatial location in order to construct an image with a holistic structure (e.g. the digit “7” in our case).
H. The upper subnetworks
In the visualization of intermediate layers (Figure 11), the intermediate layers actually represent the “lower” subnetwork from the inputs.
Here we investigate the “upper” subnetwork.
Thanks again to the spatial structure of ConvNets, we can skip the lower layers and feed the test patterns directly to the intermediate layers and still get interpretable visualizations 3 .
Figure 20 shows the results for the top-one layer from ConvNets with various depths.
A clear distinction can be found at 15-layer ConvNet, which according to Figure 3a is where the networks start to bias away from edge detector and towards the constant function.
The predictions from the final two layers of each network are visualized in Figure 21.
Figure 22 focuses on the 20-layer ConvNet that learns the constant map, and visualize the upper 3 layers, 6 layers and 10 layers, respectively.
In particular, the last visualization shows that the 20-layer ConvNet is already starting to construct the digit “7” from nowhere when using only the upper half of the model.
H. Varying training factors
The study on the effects of various training hyperparameters on the inductive bias of trained ConvNets is briefly presented in Section 3.
The full results are shown in this appendix.
layer index 0.
weight      2.
layer index 0.
bias
Convolution channel depths Figure 4b in Section 3 shows that the learned identity function is severely corrupted when only 3 channels are used in the 5-layer ConvNets.
Figure 23 presents the correlation to the constant and the identity function when different numbers of convolution channels are used.
The heatmap is consistent with the visualizations, showing that the 5-layer ConvNet fails to approximate the identity function when only three channels are used in each convolution layer.
Furthermore, Figure 24 visualize the predictions of trained 3-channel ConvNets with various depths.
The 3-channel
ConvNets beyond 8 layers fail to converge during training.
The 5-layer and the 7-layer ConvNets implement functions biased towards edge-detecting or countour-finding.
But the 6-layer and the 8-layer ConvNets demonstrate very different biases.
The potential reason is that with only a few channels, the random initialization does not have enough randomness to smooth out “unlucky” bad cases.
Therefore, the networks have higher chance to converge to various corner cases.
Figure 25 and Figure 26 compare the random initialization with the converged network for a 3-channel
ConvNet and a 128-channel ConvNet.
From the visualizations of the intermediate layers, the 128-channel ConvNet already behave more smoothly than the 3-channel ConvNet at initialization.
Convolution filter sizes Figure 27 and Figure 28 illustrate the inductive bias with varying convolution filter size from 5 × 5 to 57 × 57.
The visualization shows that the predictions become more and more blurry as the filter sizes grow.
The heatmaps, especially the correlation to the identity function, are not as helpful in this case as the correlation metric is not very good at distinguishing images with different levels of blurry.
With extremely large filter sizes that cover the whole inputs, the ConvNets start to bias towards the constant function.
Note our training inputs are of size 28 × 28, so 29 × 29 filter size allows all the neurons to see no less than half of the spatial domain from the previous layer.
57 × 57 receptive fields centered at any location within the image will be able to see the whole previous layer.
On the other hand, the repeated application of the same convolution filter through out the spatial domain is still used (with very large boundary paddings in the inputs).
So the ConvNets are not trivially doing the same computation as
FCNs.
Initialization schemes In transfer learning, it is well known that initializing with pre-trained network weights could affect the inductive bias of trained models.
On the other hand, different random initialization schemes are mainly proposed to help with optimization by maintaining information flow or norms of representations at different depths.
It turns out that they also strongly affect the inductive bias of the trained models.
Figure 29 visualizes the different layer index 0.
weight      2.
layer index 0.
bias      inductive biases.
Let f i , f o be the fan in and fan out of the layer being initialized.
We tested the following commonly used initialization schemes: 1) default: N( 0 , σ 2 = 1 (cid:14) ( f i f o )) ; 2) Xavier (a.k.a.
Glorot) init [13]: N( 0 , σ 2 = 2 (cid:14) ( f i + f o )) ; 3) Kaiming init [15]: N( 0 , σ 2 = 2 (cid:14) f i ) .
Variations with uniform distributions instead of Gaussian distributions are also evaluated.
All initialization schemes bias toward the identity function for shallow networks.
But Kaiming init produces heavy artifacts on test predictions.
For the bias towards the constant function in deep networks, Xavier init behaves similarly to the default init scheme, though more layers are needed to learn a visually good identity function.
On the other hand, the corresponding results from the Kaiming init is less interpretable.
[Figure]
Optimizers First order stochastic optimizers are dorminately used in deep learning due to the huge model sizes and dataset sizes.
To improve convergence speed, various adaptive methods are introduced [11, 17, 14].
It is known that those methods lead to worse generalization performances in some applications (e.g. Wu et al.
But they are extremely popular in practice due to the superior convergence speed and easier hyper-parameter tuning than the vanilla SGD.
We compare several popular optimizers in our framework, and confirm that different optimizers find different global minimizers, and those minimizers show drastically different inductive biases on test examples, as shown in Figure 30.
Some optimizer requires a smaller base learning rate to avoid parameter exploding during training.
In particular, we use base learning rate 0.001 for Adagrad and Adamax, 0.0001 for Adam and RMSprop.
For comparison, we also include results from SGD with those corresponding base learning rates.
I Correlation vs MSE
Figure 31, Figure 32 and Figure 33 can be compared to their corresponding figures in the main text.
The figures here are plotted with the MSE metric between the prediction and the groundtruth, while the figures in the main text uses the correlation metric.
Each corresponding pair of plots are overall consistent.
But the correlation plots show the patterns more clearly and has a fixed value range of [0, 1] that is easier to interpret.
i n p u t         i n p u t t o pt o pi n p u t t
o p
1024 # c hnn correlation to constant function 0.
rho: train-test correlation  1024 # c hnn correlation to identity function 0.
J Learning with multiple examples
We focus on learning from a single example in this paper because the two extreme inductive biases of “the identity map” and “the constant map” can be precisely defined and tested against.
When training with multiple examples, the constant map is no longer a viable solution to the optimization problem.
Nevertheless, we can still qualitatively evaluate the inductive bias via visualization of predictions.
Figure 34 shows the results on various networks trained with two examples.
In particular, for networks that are known to overfit to the constant map when trained with one example (e.g. fully connected networks with 9 hidden layers, and 20-layer convolutional networks) demonstrate similar overfitting behavior.
In this case, a proper definition of “constant map” no longer holds, as the network perfectly reconstruct each individual digit from the training set.
On the test set, a notion of memorization can be
[Figure] [Figure]
recognized as always predicting a mixture of the training samples.
Note sometimes the prediction is biased more towards one of the training samples depending on the input patterns.
Figure 35 shows the results with three examples, with similar observations.
Figure 36 shows the situation when training with the full MNIST training set (60k examples).
In this case, even the deepest convolutional networks we tried successfully learn the identity function.
Fully connected networks learn the identity function on the manifold of digit inputs, but still cannot reconstruct meaningful results on other test patterns.
[Figure] [Figure]
v
[Figure]
il
[Figure]
r
[Figure]
i
[Figure] [Figure]
[Figure] [Figure]
d
[Figure] [Figure]
[Figure] [Figure]
[Figure]
C o n v
N e t
D e p t h mse to constant function 0.
[Figure]
[Figure] [Figure]
[Figure] [Figure]