Original: Redundant representations help generalization in wide neural networks
Revised: Redundant representations help generalization in wide neural networks

Original: Abstract
Revised: Abstract

Original: Deep neural networks (DNNs) defy the classical bias-variance trade-off: adding parameters to a DNN that interpolates its training data will typically improve its generalization performance.
Revised: Deep neural networks (DNNs) defy the classical bias-variance trade-off: adding parameters to a DNN that interpolates its training data will typically improve its generalization performance.

Original: Explaining the mechanism behind this “benign overfitting” in deep networks remains an outstanding challenge.
Revised: Explaining the mechanism behind this “benign overfitting” in deep networks remains an outstanding challenge.

Original: Here, we study the last hidden layer representations of various state-of-the-art convolutional neural networks and find that if the last hidden representation is wide enough, its neurons tend to split into groups that carry identical information, and differ from each other only by statistically independent noise.
Revised: Here, we study the last hidden layer representations of various state-of-the-art convolutional neural networks and find that if the last hidden representation is wide enough, its neurons tend to split into groups that carry identical information, and differ from each other only by statistically independent noise.

Original: The number of such groups increases linearly with the width of the layer, but only if the width is above a critical value.
Revised: The number of such groups increases linearly with the width of the layer, but only if the width is above a critical value.

Original: We show that redundant neurons appear only when the training process reaches interpolation and the training error is zero.
Revised: We show that redundant neurons appear only when the training process reaches interpolation and the training error is zero.

Original:  Introduction
Revised:  Introduction

Original: Deep neural networks (DNN) routinely have enough parameters to achieve zero training error, even with random labels [1, 2].
Revised: Deep neural networks (DNN) routinely have enough parameters to achieve zero training error, even with random labels [1, 2].

Original: In defiance of the classical bias-variance trade-off, the performance of these interpolating classifiers continuously improves as the number of parameters increases well beyond the number of training samples [3–6].
Revised: In defiance of the classical bias-variance trade-off, the performance of these interpolating classifiers continuously improves as the number of parameters increases well beyond the number of training samples [3–6].

Original: Despite recent progress in describing the implicit bias of stochastic gradient descent towards “good” minima [7–12] and the detailed analysis of solvable models of learning [13–21], the mechanisms underlying this “benign overfitting” [22] in deep neural networks remain unclear, especially since their loss landscape contains “bad” local minima and SGD can reach them [23]
Revised: Despite recent progress in describing the implicit bias of stochastic gradient descent towards “good” minima [7–12] and the detailed analysis of solvable models of learning [13–21], the mechanisms underlying this “benign overfitting” [22] in deep neural networks remain unclear, especially since their loss landscape contains “bad” local minima and SGD can reach them [23]

Original: In this paper we describe a phenomenon in wide DNNs that offers a possible mechanism for benign overfitting.
Revised: In this paper we describe a phenomenon in wide DNNs that offers a possible mechanism for benign overfitting.

Original: We illustrate this mechanism in Fig.
Revised: We illustrate this mechanism in Fig.

Original: 1 for a family of increasingly wide DenseNet40s [24] on CIFAR10 [25].
Revised: 1 for a family of increasingly wide DenseNet40s [24] on CIFAR10 [25].

Original: For simplicity, we refer to the width W of last hidden representation as the width the network.
Revised: For simplicity, we refer to the width W of last hidden representation as the width the network.

Original: The blue line in Fig.
Revised: The blue line in Fig.

Original: 1 shows that the average classification error ( error ) approaches the performance of a large ensemble of networks ( error ∞ ) [21] as we increase the network width W .
Revised: 1 shows that the average classification error ( error ) approaches the performance of a large ensemble of networks ( error ∞ ) [21] as we increase the network width W .

Original: In agreement with [26], we find that the performance of these DenseNets improves continuously with width.
Revised: In agreement with [26], we find that the performance of these DenseNets improves continuously with width.

Original: For widths greater than 350, the networks are wide enough to reach zero training error (see
Revised: For widths greater than 350, the networks are wide enough to reach zero training error (see

Original: Fig S3-c in Sec A.2) and, interestingly, their test error decays approximately as W − 1 / 2 .
Revised: Fig S3-c in Sec A.2) and, interestingly, their test error decays approximately as W − 1 / 2 .

Original: Our goal is to understand how the error of the network can keep decaying beyond the interpolation threshold, and why it decays as W − 1 / 2 .
Revised: Our goal is to understand how the error of the network can keep decaying beyond the interpolation threshold, and why it decays as W − 1 / 2 .

Original: We make our key observation performing the following experiment: we randomly select a number w c of neurons from the last hidden layer of the widest DenseNet and remove all the other neurons from that layer as well as their connections (Fig.
Revised: We make our key observation performing the following experiment: we randomly select a number w c of neurons from the last hidden layer of the widest DenseNet and remove all the other neurons from that layer as well as their connections (Fig.

Original: S3-a).
Revised: S3-a).

Original: We then evaluate the performance of this “chunk” of w c neurons, without retraining the network.
Revised: We then evaluate the performance of this “chunk” of w c neurons, without retraining the network.

Original: The orange profile of Fig.
Revised: The orange profile of Fig.

Original: S3-a shows the test error of chunks of varying sizes.
Revised: S3-a shows the test error of chunks of varying sizes.

Original: There are two regimes: for small chunks, the error decays faster than w − 1 / 2 c , while beyond a critical chunk size w ∗ c (shaded area), the error of a chunk of w c neurons is roughly the same as the one of a full network with w c neurons.
Revised: There are two regimes: for small chunks, the error decays faster than w − 1 / 2 c , while beyond a critical chunk size w ∗ c (shaded area), the error of a chunk of w c neurons is roughly the same as the one of a full network with w c neurons.

Original: Furthermore, the error of the chunks decays with the same power-law w − 1 / 2 c beyond this critical chunk size.
Revised: Furthermore, the error of the chunks decays with the same power-law w − 1 / 2 c beyond this critical chunk size.

Original: The decay rate of − 1 / 2 suggests that in this regime chunks of w c neurons can be thought of as statistically independent estimators of the same features of the data, differing only by small, uncorrelated noise.
Revised: The decay rate of − 1 / 2 suggests that in this regime chunks of w c neurons can be thought of as statistically independent estimators of the same features of the data, differing only by small, uncorrelated noise.

Original: In other words, beyond the critical width w ∗ c , the final hidden representation of an input in a trained, wide DNN becomes highly redundant.
Revised: In other words, beyond the critical width w ∗ c , the final hidden representation of an input in a trained, wide DNN becomes highly redundant.

Original: This motivates a possible mechanism for benign overfitting, schematically portrayed in Fig.
Revised: This motivates a possible mechanism for benign overfitting, schematically portrayed in Fig.

Original: S3-c: as the network becomes wider, additional neurons are first used to learn new features of the data.
Revised: S3-c: as the network becomes wider, additional neurons are first used to learn new features of the data.

Original: Beyond the critical width w ∗ c , additional neurons in the final layer don’t fit to new features in the data, and hence over-fit; instead, they make a copy, or a clone , of a feature that is already part of the final representation.
Revised: Beyond the critical width w ∗ c , additional neurons in the final layer don’t fit to new features in the data, and hence over-fit; instead, they make a copy, or a clone , of a feature that is already part of the final representation.

Original: The last layer thus splits into more and more clones as the networks grows wider as we illustrate at the bottom of Fig.
Revised: The last layer thus splits into more and more clones as the networks grows wider as we illustrate at the bottom of Fig.

Original: The accuracy of these wide networks then improves with their width because the network implicitly averages over an increasing number of clones in its representations to make its prediction.
Revised: The accuracy of these wide networks then improves with their width because the network implicitly averages over an increasing number of clones in its representations to make its prediction.

Original: This paper provides a quantitative analysis of this phenomenon on various datasets and architectures.
Revised: This paper provides a quantitative analysis of this phenomenon on various datasets and architectures.

Original: Our main findings can be summarized as follows:
Revised: Our main findings can be summarized as follows:

Original: A chunk of w c random neurons of the last hidden representation of a wide neural network predicts the output with an error that decays as w − 1 / 2 c if the layer is wide enough and w c is large enough.
Revised: A chunk of w c random neurons of the last hidden representation of a wide neural network predicts the output with an error that decays as w − 1 / 2 c if the layer is wide enough and w c is large enough.

Original: In this regime, we call the chunk a “clone”; 2.
Revised: In this regime, we call the chunk a “clone”; 2.

Original: Clones can be linearly mapped one to another, or to the full representation, with an error which can be described as uncorrelated random noise.
Revised: Clones can be linearly mapped one to another, or to the full representation, with an error which can be described as uncorrelated random noise.

Original: Clones appear if the training error is zero and the model is trained with weight decay.
Revised: Clones appear if the training error is zero and the model is trained with weight decay.

Original: If training is stopped too early or if the training is performed without regularization, 1. and 2.
Revised: If training is stopped too early or if the training is performed without regularization, 1. and 2.

Original: do not take place, even if the last representation is very wide.
Revised: do not take place, even if the last representation is very wide.

Original:  Methods 2.1 Neural network architectures
Revised:  Methods 2.1 Neural network architectures

Original: We report experimental results obtained with several architectures (fully connected networks, WideResNet-28, DenseNet40, ResNet50) and data sets (CIFAR10/100 [25], ImageNet [27]).
Revised: We report experimental results obtained with several architectures (fully connected networks, WideResNet-28, DenseNet40, ResNet50) and data sets (CIFAR10/100 [25], ImageNet [27]).

Original: We train all the networks with SGD with momentum and weight decay following the standard practice to set the relevant hyperparameters.
Revised: We train all the networks with SGD with momentum and weight decay following the standard practice to set the relevant hyperparameters.

Original: We give detailed information on our training setups in A.1.
Revised: We give detailed information on our training setups in A.1.

Original: All our experiments are run on Volta V100 GPUs.
Revised: All our experiments are run on Volta V100 GPUs.

Original: In the following paragraphs, we describe how we vary the width W of the models.
Revised: In the following paragraphs, we describe how we vary the width W of the models.

Original: Fully-connected networks on MNIST.
Revised: Fully-connected networks on MNIST.

Original: We train a fully-connected network to classify the parity of the MNIST digits [28] (pMNIST) following the protocol of [21].
Revised: We train a fully-connected network to classify the parity of the MNIST digits [28] (pMNIST) following the protocol of [21].

Original: MNIST digits are projected on the first ten principal components, which are then used as inputs of a five-layer fully-connected network (FC5).
Revised: MNIST digits are projected on the first ten principal components, which are then used as inputs of a five-layer fully-connected network (FC5).

Original: The four hidden representations have the same width W and the output is a real number whose sign is the predictor of the parity of the input digit.
Revised: The four hidden representations have the same width W and the output is a real number whose sign is the predictor of the parity of the input digit.

Original: Wide-ResNet-28 and DenseNet40 on CIFAR10/100.
Revised: Wide-ResNet-28 and DenseNet40 on CIFAR10/100.

Original: We train CIFAR10 and CIFAR100 on family of Wide-ResNet-28 [26] (WR28).
Revised: We train CIFAR10 and CIFAR100 on family of Wide-ResNet-28 [26] (WR28).

Original: The number W of last hidden neurons in a WR28- n is 64 · n , obtained after average pooling the last 64 · n channels of the network.
Revised: The number W of last hidden neurons in a WR28- n is 64 · n , obtained after average pooling the last 64 · n channels of the network.

Original: In our experiments we also analyze two narrow versions of the standard WR28-1 which are not typically used in the literature.
Revised: In our experiments we also analyze two narrow versions of the standard WR28-1 which are not typically used in the literature.

Original: We name them WR28-0.25 and WR28-0.5 since they have 1/4 and 1/2 of the number of channels of WR28-1.
Revised: We name them WR28-0.25 and WR28-0.5 since they have 1/4 and 1/2 of the number of channels of WR28-1.

Original: Our implementation of DenseNet40 follows the DenseNet40-BC variant [24].
Revised: Our implementation of DenseNet40 follows the DenseNet40-BC variant [24].

Original: We vary the number of input channels c in { 8 , 16 , 32 , 64 , 128 , 256 } .
Revised: We vary the number of input channels c in { 8 , 16 , 32 , 64 , 128 , 256 } .

Original: The number W of last hidden features of this architecture is 5 . 5 · c .
Revised: The number W of last hidden features of this architecture is 5 . 5 · c .

Original: ResNet50 on ImageNet.
Revised: ResNet50 on ImageNet.

Original: We modify the ResNet50 architecture [29] by multiplying by a constant factor c ∈ { 0 . 25 , 0 . 5 , 1 , 2 , 4 } the number of channels of all the layers after the input stem.
Revised: We modify the ResNet50 architecture [29] by multiplying by a constant factor c ∈ { 0 . 25 , 0 . 5 , 1 , 2 , 4 } the number of channels of all the layers after the input stem.

Original: When c = 2 our networks differ from the standard Wide-ResNet50-2 since [26] only double the number of channels of the bottleneck of each ResNet block.
Revised: When c = 2 our networks differ from the standard Wide-ResNet50-2 since [26] only double the number of channels of the bottleneck of each ResNet block.

Original: As a consequence in our implementation the number of features after the last pooling layer is W = 2048 · c while in [26] W is fixed to 2048.
Revised: As a consequence in our implementation the number of features after the last pooling layer is W = 2048 · c while in [26] W is fixed to 2048.

Original: 2.2 Analytical methods
Revised: 2.2 Analytical methods

Original: Reconstructing the wide representation from a smaller chunk.
Revised: Reconstructing the wide representation from a smaller chunk.

Original: To assess the predictive power of  the chunked representations, we search for thebest linear map A , of dimensions W × w  , able to minimize the squared difference ( x ( W ) − ˆ x ( W ) ) 2 between the W activations of the full layer representation ( x ( W ) ) and the activations predicted from a chunk of size w , 
Revised: To determine how well a subset of w neurons can reconstruct the full representation of size W we search for the W × w linear map A , able to minimize the squared difference ( x ( W ) − ˆ x ( W ) ) 2 between the W activations of the full layer representation, x ( W ) , and the activations predicted from the chunk of size w , ˆ x ( W ) :

Original: [Equation]
Revised: [Equation]

Original: This least-squares problem is solved with ridge regression [30] with regularization set to 10 − 8 , and we use the R 2 coefficient of the fit to measure the predictive power of a given chunk size.
Revised: This least-squares problem is solved with ridge regression [30] with regularization set to 10 − 8 , and we use the R 2 coefficient of the fit to measure the predictive power of a given chunk size.

Original: The multi-output R 2 value is computed as an average of the W single-output R 2 values corresponding to the different coordinates (full layer activations), weighted by the variance of each coordinate.
Revised: The Rvalue is computed as an average of the single-activations R 2 values corresponding to the W output coordinates of the fit, weighted by the variance of each coordinate.

Original: We further compute the W × W covariance matrix C ij of the residuals of this fit, and from C ij we obtain the correlation matrix as
Revised: We further compute the W × W covariance matrix C ij of the residuals of this fit, and from C ij we obtain the correlation matrix as:

Original: [Equation]
Revised: [Equation]

Original: with a small regularization in the denominator to avoid instabilities when the standard deviation of the residuals falls below machine precision.
Revised: with a small regularization in the denominator to avoid instabilities when the standard deviation of the residuals falls below machine precision.

Original: To quantify  the independence of the chunk representations wetake the averageof the absolute values of the non-diagonal entries of the correlation matrix ρ ij .
Revised: To quantify how much the errors of the fit are correlated, we average the absolute values of the non-diagonal entries of the correlation matrix ρ ij .

Original: For short we refer to this quantity as a ‘mean correlation’.
Revised: For short, we refer to this quantity as a ‘mean correlation’.

Original: Reproducibility.
Revised: Reproducibility.

Original: We provide code to reproduce our experiments and our analysis online at https: //github.com/diegodoimo/redundant_representation .
Revised: We provide code to reproduce our experiments and our analysis online at https: //github.com/diegodoimo/redundant_representation .

Original:  Results
Revised:  Results

Original: The test error of chunks of w c neurons of the final representation asymptotically scales as w − 1 / 2 c .
Revised: The test error of chunks of w c neurons of the final representation asymptotically scales as w − 1 / 2 c .

Original: The mechanism we propose is inspired by the following experiment: we compute the test accuracy of models obtained by selecting a random subset of w c neurons from the final hidden representation of a wide neural network.
Revised: The mechanism we propose is inspired by the following experiment: we compute the test accuracy of models obtained by selecting a random subset of w c neurons from the final hidden representation of a wide neural network.

Original: We consider three different data sets (pMNIST, CIFAR10 and 
Revised: We consider three different data sets pMNIST trained on a fully connected network, CIFAR10 and CIFAR100 trained on convolutional network.

Original: CIFAR100) and trained networks of width W = 512 for pMNIST and CIFAR10, and W = 1024 for CIFAR100.
Revised: The width W of the architecture is 512 for pMNIST and CIFAR10, and W = 1024 for CIFAR100 (see Sec. 2.1).

Original: In all these cases, W is large enough to be firmly in the regime where the accuracy of the networks scales (approximately) as W − 1 / 2 (see Fig. 2).
Revised: In all these cases, W is large enough to be firmly in the regime where the accuracy of the networks scales (approximately) as W − 1 / 2 (see Fig. 2).

Original: We select w c neurons at random
Revised: We select w c neurons at random

Original: and we compute the test accuracy of a network in which we set to zero the activation of all the other w − w c neurons.
Revised: and we compute the test accuracy of a network in which we set to zero the activation of all the other w − w c neurons.

Original: Importantly, we do not fine-tune the weights after selecting the w c neurons: all the parameters are left unchanged, except that the activations of the “killed” neurons are not used to compute the logits.
Revised: Importantly, we do not fine-tune the weights after selecting the w c neurons: all the parameters are left unchanged, except that the activations of the “killed” neurons are not used to compute the logits.

Original: We take 500 random samples of neurons for each chunk width w c .
Revised: We take 500 random samples of neurons for each chunk width w c .

Original: In Fig.
Revised: In Fig.

Original: 3 we plot the test error of the chunked models as a function of w c (orange lines).
Revised: 3 we plot the test error of the chunked models as a function of w c (orange lines).

Original: The behavior is similar in all three networks: the test error of the chunks decays as w − 1 / 2 c for chunks that are larger than a critical value w ∗ c , which depends on the dataset and architecture used.
Revised: The behavior is similar in all three networks: the test error of the chunks decays as w − 1 / 2 c for chunks that are larger than a critical value w ∗ c , which depends on the dataset and architecture used.

Original: This decay follows the same law observed for full networks of the same width (Fig. 2).
Revised: This decay follows the same law observed for full networks of the same width (Fig. 2).

Original: This implies that a model obtained by selecting a random chunk of w c > w ∗ c neurons from a wide final representation behaves similarly to a full network of width W = w c .
Revised: This implies that a model obtained by selecting a random chunk of w c > w ∗ c neurons from a wide final representation behaves similarly to a full network of width W = w c .

Original: Furthermore, a decay with rate − 1 / 2 suggests that the final representation of the wide networks can be thought of as a collection of statistically independent estimates of a finite set of data features relevant for classification.
Revised: Furthermore, a decay with rate − 1 / 2 suggests that the final representation of the wide networks can be thought of as a collection of statistically independent estimates of a finite set of data features relevant for classification.

Original: Adding neurons to the chunk hence reduces their prediction error in the same way an additional measurement reduces the measurement uncertainty, leading to the − 1 / 2 decay.
Revised: Adding neurons to the chunk hence reduces their prediction error in the same way an additional measurement reduces the measurement uncertainty, leading to the − 1 / 2 decay.

Original: At w c smaller than w ∗ c instead, the test error of the chunked models decays faster than w − 1 / 2 c in all the cases we considered, including the DenseNet architecture trained on CIFAR10 shown in Fig.
Revised: At w c smaller than w ∗ c instead, the test error of the chunked models decays faster than w − 1 / 2 c in all the cases we considered, including the DenseNet architecture trained on CIFAR10 shown in Fig.

Original: 1-b.
Revised: 1-b.

Original: In this regime, adding neurons to the final representation improves the quality of the model significantly quicker than it would in independently trained models of the same width (see Fig.
Revised: In this regime, adding neurons to the final representation improves the quality of the model significantly quicker than it would in independently trained models of the same width (see Fig.

Original: 1 for a pictorial representation of this process).
Revised: 1-c for a pictorial representation of this process).

Original: We call chunks of neurons of size w c ≥ w ∗ c clones .
Revised: We call chunks of neurons of size w c ≥ w ∗ c clones .

Original: In the following we characterize more precisely the properties of the clones.
Revised: In the following we characterize more precisely the properties of the clones.

Original: Clones interpolate the training data.
Revised: Clones interpolate the training data.

Original: Awell trained deep network often represents the salient features of the data set well enough to achieve (close to) zero classification error on the training data.
Revised: A trained deep network often represents the salient features of the data set well enough to achieve (close to) zero classification error on the training data.

Original: In the top panels of Fig. 4, we show that wide networks are able to interpolate their training set also using just a subset of w c > w ∗ c random neurons: the dark orange profiles show that when the size of a chunk is greater than w ∗ c ∼ 50 for pMNIST, 100 for CIFAR10 and 200 for CIFAR100, the predictive accuracy on the training set remains almost 100%.
Revised: In the top panels of Fig. 4, we show that wide networks can interpolate their training set also using just a subset of w c > w ∗ c random neurons: the dark orange profiles show that when the size of a chunk is greater than w ∗ c ∼ 50 for pMNIST, 100 for CIFAR10 and 200 for CIFAR100, the predictive accuracy on the training set remains almost 100%.

Original: The minimal size of a clone w ∗ c , can be identified with the minimal number of neurons required to interpolate the training set.
Revised: The minimal size of a clone w ∗ c , can be identified with the minimal number of neurons required to interpolate the training set.

Original: Beyond w ∗ c , the neurons of the final representation become redundant since the training error remains (close to) zero even after removing neurons from it.
Revised: Beyond w ∗ c , the neurons of the final representation become redundant since the training error remains (close to) zero even after removing neurons from it.

Original: The number of distinct clones in a network of width W is n = W/w ∗ c .
Revised: The number of distinct clones in a network of width W is n = W/w ∗ c .

Original: If distinct clones provide independent measures of the same salient features of the data, the  error decays approximately as n − 1 / 2 or equivalently W − 1 / 2 .
Revised: If distinct clones provide independent measures of the same salient features of the data, the test error decays approximately as n − 1 / 2 or equivalently W − 1 / 2 .

Original: In the following we will indeed see that distinct clones differ from each other by uncorrelated random noise.
Revised: In the following we will indeed see that distinct clones differ from each other by uncorrelated random noise.

Original: Clones reconstruct the full representation almost perfectly.
Revised: Clones reconstruct the full representation almost perfectly.

Original: From a geometrical perspective, the important features of the final representation correspond to directions in which the data landscape shows large variations [31].
Revised: From a geometrical perspective, the important features of the final representation correspond to directions in which the data landscape shows large variations [31].

Original: A clone is a chunk that is wide enough to encode almost exactly these directions (since its training error is almost zero), but using much less neurons than the full final representation.
Revised: A clone is a chunk that is wide enough to encode almost exactly these directions (since its training error is almost zero), but using much less neurons than the full final representation.

Original: We analyze this aspect by performing a linear reconstruction of the W activations of the last layer of the widest network starting from a random chunk of w c activations using ridge regression with a small regularization penalty according to Eq.
Revised: We analyze this aspect by performing a linear reconstruction of the W activations of the last hidden representation of the widest network starting from a random subset of w c activations using ridge regression with a small regularization penalty according to Eq.

Original: The blue profiles in Fig.
Revised: The blue profiles in Fig.

Original: 4-(d,e,f), show the R 2 coefficient of fit as a function of the chunk size w c for pMNIST (left), CIFAR (center), CIFAR100 (right).
Revised: 4-(d,e,f), show the R 2 coefficient of fit as a function of the chunk size w c for pMNIST (left), CIFAR10 (center), CIFAR100 (right).

Original: When w c is very small, say below 6 for pMNIST, 20 for CIFAR and 60 for CIFAR100, the R 2 coefficient grows almost linearly with w c .
Revised: When w c is very small, say below 6 for pMNIST, 20 for CIFAR10 and 60 for CIFAR100, the R 2 coefficient grows almost linearly with w c .

Original: In this regime, adding a randomly chosen activation from the full representation to the chunk increases substantially R 2 .
Revised: In this regime, adding a randomly chosen activation from the full representation to the chunk increases substantially R 2 .

Original: When w c becomes larger R 2 reaches almost one.
Revised: When w c becomes larger R 2 reaches almost one.

Original: This transition happens when w c is still much smaller than W and correspond approximately to the regime in which the test error starts scaling with the inverse square root of w c (see Fig. 3).
Revised: This transition happens when w c is still much smaller than W and correspond approximately to the regime in which the test error starts scaling with the inverse square root of w c (see Fig. 3).

Original: The almost perfect reconstruction of the original data landscape with few neurons can be seen as a consequence of the low intrinsic dimension (ID) of the representation [32].
Revised: The almost perfect reconstruction of the original data landscape with few neurons can be seen as a consequence of the low intrinsic dimension (ID) of the representation [32].

Original: The ID of the widest representations gives a lower bound on the number of coordinates required to describe the data manifold, and hence on the neurons that a chunk needs in order to have the same classification accuracy as the whole representation.
Revised: The ID of the widest representations gives a lower bound on the number of coordinates required to describe the data manifold, and hence on the neurons that a chunk needs in order to have the same classification accuracy as the whole representation.

Original: The ID of the last hidden representation is 2 in pMNIST, 12 in CIFAR10, 14 in CIFAR100, numbers which are much lower than w  c , the width at which a chunk can be considered a clone.
Revised: The ID of the last hidden representation is 2 in pMNIST, 12 in CIFAR10, 14 in CIFAR100, numbers which are much lower than w ∗ c , the width at which a chunk can be considered a clone.

Original: Clones differ from each other by uncorrelated random noise.
Revised: Clones differ from each other by uncorrelated random noise.

Original: When w c > w ∗ c the small residual difference between the  representationchunks and the full representation can be approximately described as statistically independent random noise.
Revised: When w c > w ∗ c the small residual difference between the chunked representation and the full representation can be approximately described as statistically independent random noise.

Original: The green profile ofFig.
Revised: The green profiles of the bottom panels of

Original:  4,bottom panels, show the meanabsolute non-diagonal correlation of the residuals of the linear fit, a measure which indicates the level of correlation of the chunk representations and the full representation (see Methods).
Revised: Fig. 4, show the mean correlation of the residuals of the linear fit (see Sec. 2.2), a measure which indicates the level of correlation of the chunk representations and the full representation.

Original: Below w ∗ c , the residuals are not only large, but also significantly correlated, since they are related to relevant features of the data that are not covered by the neurons of the chunk.
Revised: Below w ∗ c , the residuals are not only large, but also significantly correlated, since they are related to relevant features of the data that are not covered by the neurons of the chunk.

Original: As the chunk width increases above w ∗ c , the correlation between residuals drops basically to zero.
Revised: As the chunk width increases above w ∗ c , the correlation between residuals drops basically to zero.

Original: Therefore, in networkswhich are wider than w ∗ c any two chunks of equal size w c > w ∗ c can be effectively considered as equivalent copies, or clones, of the same representation, differing only by a small and non-correlated noise, consistently with the scaling law of the error shown in Fig.
Revised: Therefore, in networks wider than w ∗ c any two chunks of equal size w c > w ∗ c can be effectively considered as equivalent copies, or clones, of the same representation (that of the full layer), differing only by a small and non-correlated noise, consistently with the scaling law of the error shown in Fig.

Original: The dynamics of training.
Revised: The dynamics of training.

Original: In the previous paragraphs we set forth evidence in support of the hypothesis that large chunks of the final representation of wide DNNs behave approximately like an ensemble of independent measures of the full feature space.
Revised: In the previous paragraphs we set forth evidence in support of the hypothesis that large chunks of the final representation of wide DNNs behave approximately like an ensemble of independent measures of the full feature space.

Original: This allowed us to interpret the decay of the test error of the full networks with the network width observed empirically in Fig.
Revised: This allowed us to interpret the decay of the test error of the full networks with the network width observed empirically in Fig.

Original: The three conditions that a chunked model satisfies in the regime in which its test error decays as w − 1 / 2 c are represented in Fig.
Revised: The three conditions that a chunked model satisfies in the regime in which its test error decays as w − 1 / 2 c are represented in Fig.

Original: 4: (i) the training error of the chunked model is close to zero; (ii) the chunked model can be used to reconstruct the full final representation with an R 2 ∼ 1 and (iii) the residuals of this reconstruction can be modeled as independent random noise.
Revised: 4: (i) the training error of the chunked model is close to zero; (ii) the chunked model can be used to reconstruct the full final representation with an R 2 ∼ 1 and (iii) the residuals of this reconstruction can be modeled as independent random noise.

Original: These three conditions are all observed at the end of the training.
Revised: These three conditions are all observed at the end of the training.

Original: We now analyze the training dynamics .
Revised: We now analyze the training dynamics .

Original: We will see that for the clones to arise, models not only need to be wide enough, but also, crucially, they need to be trained to maximise their performance.
Revised: We will see that for the clones to arise, models not only need to be wide enough, but also, crucially, they need to be trained to maximise their performance.

Original: Clones are formed in two stages, which occur at different times during training.
Revised: Clones are formed in two stages, which occur at different times during training.

Original: The first phase begins as soon as training starts: the network gradually adjusts the chunk representations in order to produce independent copies of the data manifold.
Revised: The first phase begins as soon as training starts: the network gradually adjusts the chunk representations in order to produce independent copies of the data manifold.

Original: This can be clearly observed in Fig.
Revised: This can be clearly observed in Fig.

Original: 5-a, which depicts the mean correlation between the residuals of the linear fit from the chunked to the full final representations of the network, the same quantity that we analyze in Fig.
Revised: 5-a, which depicts the mean correlation between the residuals of the linear fit from the chunked to the full final representations of the network, the same quantity that we analyze in Fig.

Original: 4-(d-e-f, green profiles),
Revised: 4-(d-e-f, green profiles),

Original: but now as a function of the training epoch.
Revised: but now as a function of the training epoch.

Original: Both Figs.
Revised: Both Figs.

Original: 4 and 5 analyze the WR28-8 on CIFAR10.
Revised: 4 and 5 analyze the WR28-8 on CIFAR10.

Original: As training proceeds, the correlations between residuals diminish gradually until epoch 160, and becomes particularly low for chunks greater than 64.
Revised: As training proceeds, the correlations between residuals diminish gradually until epoch 160, and becomes particularly low for chunks greater than 64.

Original: After epoch 160 further training does not bring  any sizeable reduction in their correlation.
Revised: After epoch 160 further training does not bring Between epoch and 180 the clones of the full network progressively achieve zero training In the same epochs, one observes a small improvement in the test error. any sizeable reduction in their correlation.

Original: At epoch 160 the full network also achieves zero error on the training set, as shown in Fig.
Revised: At epoch 160 the full network also achieves zero error on the training set, as shown in Fig.

Original: 5-b (brown) and Fig.
Revised: 5-b (brown) and Fig.

Original: 5-c (blue).
Revised: 5-c (blue).

Original: This event marks the end ofBetween the of progressively In the observes a small improvement the first phase, and the beginning of the second phase where the training error of the clones keeps decreasing while the full representation (blue) has already reached zero training error.
Revised: This event marks the end of the first phase, and the beginning of the second phase where the training error of the clones keeps decreasing while the full representation (blue) has already reached zero training error.

Original: For example, chunks of size 64 at epoch 150 have training errors comparable to the test error (dashed line of the middle panel).
Revised: For example, chunks of size 64 at epoch 150 have training errors comparable to the test error (dashed line of the middle panel).

Original: In the subsequent ∼ 20 epochs the training error of clones of size 128 and 256 reaches exactly zero, and the training error of chunks of size 64 reaches a plateau.
Revised: In the subsequent ∼ 20 epochs the training error of clones of size 128 and 256 reaches exactly zero, and the training error of chunks of size 64 reaches a plateau.

Original: Importantly, both phases improve the generalization properties of the network.
Revised: Importantly, both phases improve the generalization properties of the network.

Original: This can be seen in
Revised: This can be seen in

Original: Fig.
Revised: Fig.

Original: 5-c, which reports training and test error of the network, with the two phases highlighted.
Revised: 5-c, which reports training and test error of the network, with the two phases highlighted.

Original: The figure shows that both phases lead to a reduction in the test error, although the first phase leads by far to the greatest reduction, consistently with the fact that the greatest improvements in accuracy typically arise during the first epochs of training.
Revised: The figure shows that both phases lead to a reduction in the test error, although the first phase leads by far to the greatest reduction, consistently with the fact that the greatest improvements in accuracy typically arise during the first epochs of training.

Original: The formation of clones can be considered finished around epoch 180, when all the clones have reached almost zero error on the training set.
Revised: The formation of clones can be considered finished around epoch 180, when all the clones have reached almost zero error on the training set.

Original: After epoch 180 we also observe that the test error stops improving.
Revised: After epoch 180 we also observe that the test error stops improving.

Original: In Sec.
Revised: In Sec.

Original: A.2 we report the same analysis done on CIFAR100 (see Fig. S1) and CIFAR10 trained on a DenseNet40 (see Fig.
Revised: A.2 we report the same analysis done on CIFAR100 (see Fig. S1) and CIFAR10 trained on a DenseNet40 (see Fig.

Original: S3-(d-e-f)).
Revised: S3-(d-e-f))

Original: Clones appear only in regularized networks.
Revised: Clones appear only in regularized networks.

Original: So far in this work, we have shown only examples of networks and datasets in which representations are redundant.
Revised: So far in this work, we have shown only examples of networks and datasets in which representations are redundant.

Original: However, if the network is not regularized, some of the signatures described above don’t appear even if the width of the final representation is much larger than W ∗ (the minimum interpolating width).
Revised: However, if the network is not regularized, some of the signatures described above don’t appear even if the width of the final representation is much larger than W ∗ (the minimum interpolating width).

Original: Figure 6 shows the case of the Wide-ResNet28-8 analyzed in Fig.
Revised: Figure 6 shows the case of the Wide-ResNet28-8 analyzed in Fig.

Original: 5 trained on exactly the same dataset (CIFAR10) but without weight decay.
Revised: 5 trained on exactly the same dataset (CIFAR10) but without weight decay.

Original: As shown in Fig.6-a in the network trained without regularization (blue line) the error does not scale as w − 1 / 2 c .
Revised: As shown in Fig.6-a in the network trained without regularization (blue line) the error does not scale as w − 1 / 2 c .

Original: This, as we have seen, indicates that the last  representation cannot be split in chunks.
Revised: This, as we have seen, indicates that the last hidden representation cannot be split in clones equivalent to the full layer.

Original: Indeed, the mean correlation of the residuals of the linear map of the chunks to the full representation remains approximately constant during training (Fig.
Revised: Indeed, the mean correlation of the residuals of the linear map of the chunks to the full representation remains approximately constant during training (Fig.

Original: 6-b),
Revised: 6-b),

Original: and is always much higher then what we observed for the same architecture and dataset, when training is performed with weight decay.
Revised: and is always much higher then what we observed for the same architecture and dataset, when training is performed with weight decay.

Original: We performed a similar analysis on the DenseNet40 (see Fig. S4), observing an analogous trend.
Revised: We performed a similar analysis on the DenseNet (see Fig. S4), observing an analogous trend.

Original: Clones appear only if a network interpolates the training set: the case of ImageNet.
Revised: Clones appear only if a network interpolates the training set: the case of ImageNet.

Original: We saw that a chunk of neurons can be considered a clone if it fully captures the relevant features of the data, achieving almost zero training error (see Fig. 4).
Revised: We saw that a chunk of neurons can be considered a clone if it fully captures the relevant features of the data, achieving almost zero training error (see Fig. 4).

Original: This condition is not satisfied for most of the networks trained on ImageNet [16], therefore we do not expect to see redundant representations in this important case.
Revised: This condition is not satisfied for most of the networks trained on ImageNet [16], therefore we do not expect to see redundant representations in this important case.

Original: We verified this hypothesis by training a family of ResNet50s where we multiply all the channels of the layers after the input stem by a constant factor c ∈ { 0 . 25 , 0 . 5 , 1 , 2 , 4 } .
Revised: We verified this hypothesis by training a family of ResNet50s where we multiply all the channels of the layers after the input stem by a constant factor c ∈ { 0 . 25 , 0 . 5 , 1 , 2 , 4 } .

Original: In this manner the widest final representation we consider consists of 8192 neurons, which is four times wider than both the standard ResNet50 [29] and its wider version [26] (see Sec. 2.1).
Revised: In this manner the widest final representation we consider consists of 8192 neurons, which is four times wider than both the standard ResNet50 [29] and its wider version [26] (see Sec. 2.1).

Original: We trained all the networks following the standard protocols and achieved test errors comparable or slightly lower than those reported in the literature (see Sec. A.1).
Revised: We trained all the networks following the standard protocols and achieved test errors comparable or slightly lower than those reported in the literature (see Sec. A.1).

Original: We find that even in the case of the largest
Revised: We find that even in the case of the largest

Original: ResNet50, the top-1 error on the training set is ∼ 8% (see Fig.
Revised: ResNet50, the top-1 error on the training set is ∼ 8% (see Fig.

Original: S2a) and the network do not achieve interpolation, consistently with [16].
Revised: S2a) and the network do not achieve interpolation, as discussed also in [16].

Original: In this setting, none of the elements associated with the development of independent clones can be observed.
Revised: In this setting, none of the elements associated with the development of independent clones can be observed.

Original: The scaling of the test error of the chunks is steeper than w − 1 / 2 c (see Fig.
Revised: The scaling of the test error of the chunks is steeper than w − 1 / 2 c (see Fig.

Original: S2-b) suggesting that chunks remain significantly correlated to each other.
Revised: S2-b) suggesting that chunks remain significantly correlated to each other.

Original: Figure S2-c shows that the mean correlation of the residuals does not decrease during training, as it happens for the networks we trained on
Revised: Figure S2-c shows that the mean correlation of the residuals does not decrease during training, as it happens for the networks we trained on

Original: CIFAR10 and CIFAR100.
Revised: CIFAR10 and CIFAR100.

Original: We conclude that in a ResNet50, a representation with 8192 neurons is too narrow to encode all the relevant features redundantly on ImageNet and a chunk as large as 4096 activations is not able to reconstruct all the relevant variations of the data as it does in the cases analyzed in Sec. 3.
Revised: We conclude that in a ResNet50, a representation with 8192 neurons is too narrow to encode all the relevant features redundantly on ImageNet and a chunk as large as 4096 activations is not able to reconstruct all the relevant variations of the data as it does in the cases analyzed in Sec. 3.

Original:  Discussion
Revised:  Discussion

Original: This work is an attempt to explain the paradoxical observation that over-parameterization boosts the performance of DNNs.
Revised: This work is an attempt to explain the paradoxical observation that over-parameterization boosts the performance of DNNs.

Original: This “paradox” is not a peculiarity of DNNs: if one trains a prediction model with n parameters using the same training set, but starting from independent initial weights and receiving samples in an independent way, one can obtain, say, m models which, in suitable conditions, provide predictions of the same quantity with independent noise due to initialization,
Revised: This “paradox” is not a peculiarity of DNNs: if one trains a prediction model with n parameters using the same training set, but starting from independent initial weights and receiving samples in an independent way, one can obtain, say, m models which, in suitable conditions, provide predictions of the same quantity with independent noise due to initialization,

Original: SGD schedule, etc.
Revised: SGD schedule, etc.

Original: If one estimates the target quantity by an ensemble average, the statistical error will (ideally) scale with m − 1 / 2 , and therefore with N − 1 / 2 , where N = n m is the total number of parameters of the combined model.
Revised: If one estimates the target quantity by an ensemble average, the statistical error will (ideally) scale with m − 1 / 2 , and therefore with N − 1 / 2 , where N = n m is the total number of parameters of the combined model.

Original: This will happen even if N is much larger than the number of data.
Revised: This will happen even if N is much larger than the number of data.

Original: What is less trivial is that a DNN can accomplish this scaling within a single model, in which all the parameters are optimized collectively via the minimization of a single loss function.
Revised: What is less trivial is that a DNN can accomplish this scaling within a single model, in which all the parameters are optimized collectively via the minimization of a single loss function.

Original: Our work describes a possible mechanism at the basis of this phenomenon in the special case ofa neural networks in which the last layer is very wide.
Revised: Our work describes a possible mechanism at the basis of this phenomenon in the special case of neural networks in which the last layer is very wide.

Original: We observe that if the layer is wide enough, random subsets of its neurons can be viewed as approximately independent representations of the same data manifold (or clones).
Revised: We observe that if the layer is wide enough, random subsets of its neurons can be viewed as approximately independent representations of the same data manifold (or clones).

Original: This implies a scaling of the error with the width of the layer as W − 1 / 2 , which is qualitatively consistent with our observations.
Revised: This implies a scaling of the error with the width of the layer as W − 1 / 2 , which is qualitatively consistent with our observations.

Original: The impact of network architecture.
Revised: The impact of network architecture.

Original: The capability of a network to produce statistically independent clones is a genuine effect of the over-parametrization of the whole network as we find that redundancies appear even if the last layer width is kept constant and the width of all intermediate layers is increased (see Fig.
Revised: The capability of a network to produce statistically independent clones is a genuine effect of the over-parametrization of the whole network as we find that redundancies appear even if the last layer width is kept constant and the width of all intermediate layers is increased (see Fig.

Original: S5-a, Sec. A.2).
Revised: S5-a, Sec. A.2).

Original: At the same time, we also verified that if the network is too narrow to interpolate the training set, increasing the width of only the final representation is not sufficient to make the last layer redundant.
Revised: At the same time, we also verified that if the network is too narrow to interpolate the training set, increasing the width of only the final representation is not sufficient to make the last layer redundant.

Original: We give an example of this effect in Fig.
Revised: We give an example of this effect in Fig.

Original: S5-b, where we show that the test error of a WR28-1 on CIFAR10 does not decrease if only the width of the final representation is increased, while the rest of the architecture is kept at a constant width.
Revised: S5-b, where we show that the test error of a WR28-1 on CIFAR10 does not decrease if only the width of the final representation is increased, while the rest of the architecture is kept at a constant width.

Original: The impact of training.
Revised: The impact of training.

Original: The mechanism we described is robust to different training objectives since we trained the convolutional networks with cross-entropy loss and the fully connected networks with hinge loss.
Revised: The mechanism we described is robust to different training objectives since we trained the convolutional networks with cross-entropy loss and the fully connected networks with hinge loss.

Original: However, even for wide enough architectures, clones appear only if the training is continued until the training error reaches zero.
Revised: However, even for wide enough architectures, clones appear only if the training is continued until the training error reaches zero.

Original: In our examples, by stopping the training too early, for example when the training error is similar to the test error, the chunks of the last representation would not become entirely independent from one another, and therefore they could not be considered clones.
Revised: In our examples, by stopping the training too early, for example when the training error is similar to the test error, the chunks of the last representation would not become entirely independent from one another, and therefore they could not be considered clones.

Original: Neural scaling laws.
Revised: Neural scaling laws.

Original: Capturing the asymptotic performance of neural network via scaling laws is another active research area.
Revised: Capturing the asymptotic performance of neural network via scaling laws is an active research area.

Original: Hestness et al.
Revised: Hestness et al.

Original: [33] gave an experimental analysis of scaling laws w.r.t the training data set size in a variety of domains.
Revised: [33] gave an experimental analysis of scaling laws w.r.t the training data set size in a variety of domains.

Original: [34, 35] experimentally explored the scaling of the generalization error of deep networks with the number of parameters/data points across architectures and application domains for supervised learning, while [36] identified empirical scaling laws in generative models.
Revised: [34, 35] experimentally explored the scaling of the generalization error of deep networks with the number of parameters/data points across architectures and application domains for supervised learning, while [36] identified empirical scaling laws in generative models.

Original: Geiger et al.
Revised: Geiger et al.

Original: [21] found that the generalization error of networks trained on pMNIST without weight decay, momentum and data augmentation scales as W − 1 .
Revised: [21] found that the generalization error of networks trained on pMNIST without weight decay, momentum and data augmentation scales as W − 1 .

Original: Bahri et al.
Revised: Bahri et al.

Original: [37] showed the existence of four scaling regimes and described them theoretically in the NTK or lazy regime [38–40], where the network weights stay close to their initial values throughout training.
Revised: [37] showed the existence of four scaling regimes and described them theoretically in the NTK or lazy regime [38–40], where the network weights stay close to their initial values throughout training.

Original: We instead consider the feature learning regime, where weights move considerably as the networks learn task-relevant features.
Revised: We instead consider the feature learning regime, where weights move considerably as the networks learn task-relevant features.

Original: Moreover, we train our networks with weight decay which is unavoidable to obtain models with state-of-the-art performance.
Revised: Moreover, we train our networks with weight decay which is unavoidable to obtain models with state-of-the-art performance.

Original: Previous theoretical work mainly focused on the NTK regime and did not study the impact of weight decay on scaling laws, so we hope that our results can spark further studies on the role of this essential regularizer.
Revised: Previous theoretical work mainly focused on the NTK regime and did not study the impact of weight decay on scaling laws, so we hope that our results can spark further studies on the role of this essential regularizer.

Original: Relation to theoretical results in the mean-field regime.
Revised: Relation to theoretical results in the mean-field regime.

Original: Our empirical results also agree with recent theoretical results that were obtained for two-layer neural networks [41–46].
Revised: Our empirical results also agree with recent theoretical results that were obtained for two-layer neural networks [41–46].

Original: These works characterize the optimal solutions of two-layer networks trained on synthetic datasets with some controlled features.
Revised: These works characterize the optimal solutions of two-layer networks trained on synthetic datasets with some controlled features.

Original: In the limit of infinite training data, these optimal solutions correspond to networks where neurons in the hidden layer duplicate the key features of the data.
Revised: In the limit of infinite training data, these optimal solutions correspond to networks where neurons in the hidden layer duplicate the key features of the data.

Original: These “denoising solutions” or “distributional fixed points” were found for networks with wide hidden layers [41–44] and wide input dimension [45, 46].
Revised: These “denoising solutions” or “distributional fixed points” were found for networks with wide hidden layers [41–44] and wide input dimension [45, 46].

Original: Another point of connection with the theoretical literature is the concept of dropout stability .
Revised: Another point of connection with the theoretical literature is the concept of dropout stability .

Original: A network is said to be ϵ -dropout
Revised: A network is said to be ϵ -dropout

Original: stable if its training loss changes by less than ϵ when half the neurons are removed at random from each of its layers [47].
Revised: stable if its training loss changes by less than ϵ when half the neurons are removed at random from each of its layers [47].

Original: Dropout stability has been rigorously linked to several phenomena in neural networks, such as the connectedness of the minima of their training landscape [48, 49].
Revised: Dropout stability has been rigorously linked to several phenomena in neural networks, such as the connectedness of the minima of their training landscape [48, 49].

Original: Implicit ensembling in deep learning.
Revised: Implicit ensembling in deep learning.

Original: The success of various deep learning architectures and techniques has been linked to some form of ensembling.
Revised: The success of various deep learning architectures and techniques has been linked to some form of ensembling.

Original: [50] proposed that the performance of ResNets [29] stems from an effective ensembling of shallower networks due to the residual connections.
Revised: [50] proposed that the performance of ResNets [29] stems from an effective ensembling of shallower networks due to the residual connections.

Original: The successful dropout regularisation technique [51, 52] samples from an exponential number of “thinned” networks during training to prevent co-adaptation of hidden units.
Revised: The successful dropout regularisation technique [51, 52] samples from an exponential number of “thinned” networks during training to prevent co-adaptation of hidden units.

Original: While this can be seen as a form of (implicit) ensembling, here we observe that co-adaptation of hidden units in the form of clones occurs without dropout, and is crucial for their improving performance with width.
Revised: While this can be seen as a form of (implicit) ensembling, here we observe that co-adaptation of hidden units in the form of clones occurs without dropout, and is crucial for their improving performance with width.

Original: Recent theoretical work on random features [18–20] also suggests that ensembling and overparameterization are two sides of the same coin, at least in the lazy regime of neural networks [40], where the weights of the networks’ hidden layers don’t move much during training.
Revised: Recent theoretical work on random features [18–20] also suggests that ensembling and overparameterization are two sides of the same coin, at least in the lazy regime of neural networks [40], where the weights of the networks’ hidden layers don’t move much during training.

Original: In this paper instead, we focus on networks in the feature learning regime where weights move significantly.
Revised: In this paper instead, we focus on networks in the feature learning regime where weights move significantly.

Original: We hope that our results motivate theoretical studies along those lines in the feature learning regime.
Revised: We hope that our results motivate theoretical studies along those lines in the feature learning regime.

Original: Impact for applications.
Revised: Impact for applications.

Original: The framework introduced in this work allows verifying if a neural network is sufficiently expressive to encode multiple statistically independent representations of the same ground truth, which, we believe, is a fair proxy of model quality and robustness.
Revised: The framework introduced in this work allows verifying if a neural network is sufficiently expressive to encode multiple statistically independent representations of the same ground truth, which, we believe, is a fair proxy of model quality and robustness.

Original: In particular, we find that reaching interpolation on the training set is not necessarily detrimental for generalization, and is instead a necessary condition for developing redundancies which, in turn, reduce the test error.
Revised: In particular, we find that reaching interpolation on the training set is not necessarily detrimental for generalization, and is instead a necessary condition for developing redundancies which, in turn, reduce the test error.

Original: A Appendix A.1 Hyperparameters used and training procedures
Revised: A Appendix A.1 Hyperparameters used and training procedures

Original: Fully-connected networks on MNIST.
Revised: Fully-connected networks on MNIST.

Original: We train the fully-connected networks for 5000 epochs with stochastic gradient descent using the following hyperparameters: batch size = 256, momentum = 0.9, learning rate = 10 − 3 , weight decay = 10 − 2 .
Revised: We train the fully-connected networks for 5000 epochs with stochastic gradient descent using the following hyperparameters: batch size = 256, momentum = 0.9, learning rate = 10 − 3 , weight decay = 10 − 2 .

Original: We optimize our networks using Adam.
Revised: We optimize our networks using Adam.

Original: Wide-ResNet-28 and DenseNet40-BC on CIFAR10/100.
Revised: Wide-ResNet-28 and DenseNet40-BC on CIFAR10/100.

Original: All the models are trained forepochs with stochastic gradient descent with a batch size = 128, momentum = 0.9, and cosine annealing scheduler starting with a learning rate of 0.1.
Revised: All the models are trained forepochs with stochastic gradient descent with a batch size = 128, momentum = 0.9, and cosine annealing scheduler starting with a learning rate of 0.1.

Original: The training set is augmented with horizontal flips with 50% probability and random cropping the images padded with four pixels on each side.
Revised: The training set is augmented with horizontal flips with 50% probability and random cropping the images padded with four pixels on each side.

Original: On CIFAR10 trained on WR28 we select a weight decay equal to 5 · 10 − 4 and label smoothing magnitudes equal to 0.1 for WR28-{0.25, 0.5, 1, 2} and equal to 0 for WR28-{4, 8}.
Revised: On CIFAR10 trained on WR28 we select a weight decay equal to 5 · 10 − 4 and label smoothing magnitudes equal to 0.1 for WR28-{0.25, 0.5, 1, 2} and equal to 0 for WR28-{4, 8}.

Original: On CIFAR trained on Densenet40-BC we set select a weight decay equal to 5 · 10 − 4 and label smoothing magnitudes equal to 0.05 for all the networks On CIFAR100 trained on WR28 we set weight decays equal to {10, 7, 5, 5, 5} · 10 − 4 and label smoothing magnitudes equal to {0.1, 0.07, 0.05, 0, 0} for
Revised: On CIFAR trained on Densenet40-BC we set select a weight decay equal to 5 · 10 − 4 and label smoothing magnitudes equal to 0.05 for all the networks On CIFAR100 trained on WR28 we set weight decays equal to {10, 7, 5, 5, 5} · 10 − 4 and label smoothing magnitudes equal to {0.1, 0.07, 0.05, 0, 0} for

Original: WR28-{1, 2, 4, 8, 16} respectively.
Revised: WR28-{1, 2, 4, 8, 16} respectively.

Original: All the hyperparameters were selected with a small grid search.
Revised: All the hyperparameters were selected with a small grid search.

Original: ResNet50 on ImageNet.
Revised: ResNet50 on ImageNet.

Original: We train all the ResNet50 with mixed precision [53] for 120 epochs with a weight decay of 4 · 10 − 5 and label smoothing rate of 0.1 [54].
Revised: We train all the ResNet50 with mixed precision [53] for 120 epochs with a weight decay of 4 · 10 − 5 and label smoothing rate of 0.1 [54].

Original: The input size is 224 × 224 and the training set is augmented with random crops and horizontal flips with 50% probability.
Revised: The input size is 224 × 224 and the training set is augmented with random crops and horizontal flips with 50% probability.

Original: The per-GPU batch size is set to 128 and is halved for the widest networks to fit in the GPU memory.
Revised: The per-GPU batch size is set to 128 and is halved for the widest networks to fit in the GPU memory.

Original: The networks are trained on 8 or 16 Volta V100 GPUs so as to keep the batch size B equal to 1024.
Revised: The networks are trained on 8 or 16 Volta V100 GPUs so as to keep the batch size B equal to 1024.

Original: The learning rate is increased linearly from
Revised: The learning rate is increased linearly from

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]