Original: Are All Layers Created Equal?
Revised: Are All Layers Created Equal?

Original: Author

Original: Author

Original: Author

Original: Abstract
Revised: Abstract

Original: Understanding learning and generalization of deep architectures has been a major research objective in the recent years with notable theoretical progress.
Revised: Understanding learning with deep architectures has been a major research objective in the recent years with notable theoretical progress.

Original: A main focal point of generalization studies stems from the success of excessively large networks which defy the classical wisdom of uniform convergence and learnability.
Revised: A main focal point of those studies stems from the success of excessively large networks.

Original: We study empirically the layer-wise functional structure of overparameterized deep models.
Revised: We study empirically the layer-wise functional structure of overparameterized deep models.

Original: We provide evidence for the heterogeneous characteristic of layers.
Revised: We provide evidence for the heterogeneous characteristic of layers.

Original: To do so, we introduce the notion of (post training) re-initialization and re-randomization robustness.
Revised: To do so, we introduce the notion of (post training) re-initialization and re-randomization robustness.

Original: We show that layers can be categorized into either “robust” or “critical”.
Revised: We show that layers can be categorized into either “robust” or “critical”.

Original: In contrast to critical layers, resetting the robust layers to their initial value has no negative consequence, and in many cases they barely change throughout training.
Revised: In contrast to critical layers, resetting the robust layers to their initial value has no negative consequence, and in many cases they barely change throughout training.

Original: Our study providesfurther evidence that mere parameter counting or norm accounting is too coarse in studying generalization of deep models, and ﬂatness or robustness analysis of the model parameters needs to respect the network architectures
Revised: Our study provides evidence flatness or robustness analysis of the model parameters needs to respect the network architectures.

Original:  Introduction
Revised:  Introduction

Original: Deep neural networks have been remarkably successful in many real world machine learning applications.
Revised: Deep neural networks have been remarkably successful in many real world machine learning applications.

Original: Distilled understanding of the systems is at least as important as their state-of-the-art performance when applying them in many critical domains.
Revised: Distilled understanding of the systems is at least as important as their state-of-the-art performance when applying them in many critical domains.

Original: Recent work on understanding why deep networks perform so well in practice focused on questions such as networks’ performance under drifting or even adversarially perturbed data distribution.
Revised: Recent work on understanding why deep networks perform so well in practice focused on questions such as networks’ performance under drifting or even adversarially perturbed data distribution.

Original: Another interesting and relevant to this work is research on how we can interpret or explain the decision function of trained networks.
Revised: Another interesting and relevant to this work is research on how we can interpret or explain the decision function of trained networks.

Original: While related, this work takes a different angle as we focus on the role of the layers in trained networks and then relate the empirical results togeneralization and robustness properties.
Revised: While related, this work takes a different angle as we focus on the role of the layers in trained networks and then relate the empirical results to robustness properties.

Original: Theoretical research on the representation power of neural networks is well studied.
Revised: Theoretical research on the representation power of neural networks is well studied.

Original: It is known that a neural network with a single sufﬁciently wide hidden layer is universal approximator for continuous functions over a compact domain [11, 18, 2].
Revised: It is known that a neural network with a single sufficiently wide hidden layer is universal approximator for continuous functions over a compact domain [11, 18, 2].

Original: More recent research further examines whether deep networks can have superior representation power than shallow ones with the same number of units or edges [27, 5, 24, 35, 31, 9, 23, 29].
Revised: More recent research further examines whether deep networks can have superior representation power than shallow ones with the same number of units or edges [27, 5, 24, 35, 31, 9, 23, 29].

Original: The capacity to represent arbitrary functions with ﬁnite samples is also extensively discussed in recent work [13, 40, 26, 39].
Revised: The capacity to represent arbitrary functions with finite samples is also extensively discussed in recent work [13, 40, 26, 39].

Original: However, the constructions used in the aforementioned work for building networks approximating particular functions are typically “artiﬁcial” and are unlikely to be obtained by gradient-based learning algorithms.
Revised: However, the constructions used in the aforementioned work for building networks approximating particular functions are typically “artificial” and are unlikely to be obtained by gradient-based learning algorithms.

Original: We focus instead on empirically studying the role different layers in deep architecture take post gradient-based training.
Revised: We focus instead on empirically studying the role different layers in deep architecture take post gradient-based training.

Original: In particular, we show empirically that the layers in a deep network are not homogeneous in the role they play at representing a prediction function.
Revised: In particular, we show empirically that the layers in a deep network are not homogeneous in the role they play at representing a prediction function.

Original: Some layers are critical to forming good predictions while others are fairly robust to the assignment of their parameters along training.
Revised: Some layers are critical to forming good predictions while others are fairly robust to the assignment of their parameters along training.

Original: Moreover, depending on the capacity of the network and the complexity of the target function, gradient-based trained networks conserve the complexity by not using excess capacity.
Revised: Moreover, depending on the capacity of the network and the complexity of the target function, gradient-based trained networks conserve the complexity by not using excess capacity.

Original: The exact deﬁnition and implications on generalization are discussed in the body of the paper.

Original:  Setting
Revised:  Setting

Original: Let F D
Revised: Let  𝐷

Original: = { f θ : θ = ( θ 1 ,..., θ D )} be the space of a particular neural network architecture with D (parametric) layers.
Revised: = {𝑓 𝜃 ∶ 𝜃 = (𝜃 1 , … , 𝜃 𝐷 )} be the space of a particular neural network architecture with 𝐷 (parametric) layers.

Original: We use the term capacity to refer to properties of the entire space F D before training takes place (e.g. Rademacher complexity , VC Dimension ).
Revised: We use the term capacity to refer to properties of the entire space  𝐷 before training takes place (e.g. Rademacher complexity , VC Dimension ).

Original: The term complexity refers to properties of a single neural network f θ , often employing with notion of norm of the parameters θ and possibly normalized by empirical quantities such as the margin .[Figure]
Revised: The term complexity refers to properties of a single neural network 𝑓 𝜃 , often employing with notion of norm of the parameters 𝜃 and possibly normalized by empirical quantities such as the margin .

Original: We are interested in analyzing post-training behavior of layers of popular deep networks.
Revised: We are interested in analyzing post-training behavior of layers of popular deep networks.

Original: Such networks are typically trained using SGD with randomly sampled initial weights from pre-deﬁned distributions θ 0 d ∼ P d , typically depending  on the type, fan-in, and fan-out of of each layer.
Revised: Such networks are typically trained using SGD with randomly sampled initial weights from pre-defined distributions 𝜃 0𝑑 ∼  𝑑 , typically depending [Figure] on the type, fan-in, and fan-out of of each layer.

Original: In a deep network, the representations at a particular layer recursively depend on all the layers beneath it.
Revised: In a deep network, the representations at a particular layer recursively depend on all the layers beneath it.

Original: This complex dependency makes it challenging to isolate and inspect each layer independently in theoretical studies.
Revised: This complex dependency makes it challenging to isolate and inspect each layer independently in theoretical studies.

Original: In this paper, we introduce and use the following two empirical probes to inspect the individual layers in a trained neural network
Revised: In this paper, we introduce and use the following two empirical probes to inspect the individual layers in a trained neural network.

Original: Re-initialization After training, for a given layer d = 1,..., D , we can re-initialize the parameters through assignment θ Td ← θ 0 d , while keeping the parameters for the other layers unchanged.
Revised: Re-initialization After training, for a given layer 𝑑 = 1, … , 𝐷 , we can re-initialize the parameters through assignment 𝜃 𝑇𝑑 ← 𝜃 0𝑑 , while keeping the parameters for the other layers unchanged.

Original: The model with the parameters ( θ T 1 ,..., θ Td − 1 , θ 0 d , θ Td + 1 ,..., θ TD ) is then evaluated.
Revised: The model with the parameters (𝜃 𝑇1 , … , 𝜃 𝑇𝑑−1 , 𝜃 0𝑑 , 𝜃 𝑇𝑑+1 , … , 𝜃 𝑇𝐷 ) is then evaluated.

Original: Unless noted otherwise, we use the term performance to designate classiﬁcation error on test data.
Revised: Unless noted otherwise, we use the term performance to designate classification error on test data.

Original: The performance of a network in which layer d was re-initialized is referred to as the re-initialization robustness of layer d .
Revised: The performance of a network in which layer 𝑑 was re-initialized is referred to as the re-initialization robustness of layer 𝑑 .

Original: Note that here θ 0 d denotes the random values realized at the beginning of the training.
Revised: Note that here 𝜃 0𝑑 denotes the random values realized at the beginning of the training.

Original: More generally, for k time steps 0 = τ 1 < τ 2 < ··· < τ k − 1 < τ k = T , we can re-initialize the d -th layer by setting θ Td ← θ τ d , and obtain the re-initialization robustness of layer d after τ updates.
Revised: More generally, for 𝑘 time steps 0 = 𝜏 1 < 𝜏 2 < ⋯ < 𝜏 𝑘−1 < 𝜏 𝑘 = 𝑇 , we can re-initialize the 𝑑 -th layer by setting 𝜃 𝑇𝑑 ← 𝜃 𝜏𝑑 , and obtain the re-initialization robustness of layer 𝑑 after 𝜏 updates.

Original: Re-randomization To go one step further, we also examine re-randomization of a layer d by re-sampling random values ˜ θ d ∼ P d and evaluate the model’s performance for ( θ T 1 ,..., θ Td − 1 , ˜ θ d , θ Td + 1 ,..., θ TD ).
Revised: Re-randomization To go one step further, we also examine re-randomization of a layer 𝑑 by re-sampling random values ̃𝜃 𝑑 ∼  𝑑 and evaluate the model’s performance for (𝜃 𝑇1 , … , 𝜃 𝑇𝑑−1 , ̃𝜃 𝑑 , 𝜃 𝑇𝑑+1 , … , 𝜃 𝑇𝐷 ) .

Original: Analogously, we refer to the evaluated performance as the re-randomization robustness of layer d .
Revised: Analogously, we refer to the evaluated performance as the re-randomization robustness of layer 𝑑 .

Original: Note that there is no re-training or ﬁnetuning after re-initialization or re-randomization, and the network is evaluated directly with mixed weights.
Revised: Note that there is no re-training or finetuning after re-initialization or re-randomization, and the network is evaluated directly with mixed weights.

Original: When a network exhibits no or negligible decrease in performance after re-initializing or re-randomizing of a layer, we say that the layer is robust, and otherwise the layer is called critical.
Revised: When a network exhibits no or negligible decrease in performance after re-initializing or re-randomizing of a layer, we say that the layer is robust, and otherwise the layer is called critical.

Original:  Robustness of individual layers
Revised:  Robustness of individual layers

Original: We start by examining robustness of fully-connected networks (FCN).
Revised: We start by examining robustness of fully-connected networks (FCN).

Original: A FCN D × H consists of D fully connected layers each of which of output dimension H and ReLU activation function.
Revised: A FCN 𝐷 × 𝐻 consists of 𝐷 fully connected layers each of which of output dimension 𝐻 and ReLU activation function.

Original: The extra ﬁnal layer is a linear multiclass predictor with one output per class.
Revised: The extra final layer is a linear multiclass predictor with one output per class.

Original: As a starter, we trained an FCN 3 × 256 on the MNIST digit classiﬁcation task, and applied the re-initialization and re-randomization analysis on the trained model.
Revised: As a starter, we trained an FCN 3 × 256 on the MNIST digit classification task, and applied the re-initialization and re-randomization analysis on the trained model.

Original: The results are shown in Figure 1(a).
Revised: The results are shown in Figure 1(a).

Original: As expected, due to the intricate dependency of the classiﬁcation function on each of the layers, re-randomizing any of the layers completely disintegrate the representations and classiﬁcation accuracy drops to the level of random guessing.
Revised: As expected, due to the intricate dependency of the classification function on each of the layers, re-randomizing any of the layers completely disintegrate the representations and classification accuracy drops to the level of random guessing.

Original: However, for re-initialization, we ﬁnd that while the ﬁrst layer is very sensitive, the rest of the layers are robust to re-initializing back to their pre-training random weights.
Revised: However, for re-initialization, we find that while the first layer is very sensitive, the rest of the layers are robust to re-initializing back to their pre-training random weights.

Original: A plausible reason could be attributed to the fact that gradient norms increase during back-propagation to the point that the bottom layers are being updated more aggressively than the top ones.
Revised: A plausible reason could be attributed to the fact that gradient norms increase during back-propagation to the point that the bottom layers are being updated more aggressively than the top ones.

Original: If this was the case, we would expect 
Revised: If this was the case, we would expect a smoother transition instead of a sharp contrast at the first layer. We thus measured how distant the weights of each layer from their initialization using the normalized 2 -norm and the ∞ -norm, in Figure 1(b) and (c), respectively. It turns out the robustness to re-initialization does not obviously correlate to either of the distances.

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Table]

Original: [Table] 
Revised: [Table] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Figure]

Revised: l

Original: [Table]
Revised: [Figure]

Revised: s t

Original: [Table]
Revised: [Figure]

Revised: g e

Original: [Table]
Revised: [Figure]

Revised: e

Original: [Table]
Revised: [Figure]

Revised: b l k 2 s t a g e

Original: [Figure]
Revised: [Figure]

Original:  sb l k
Revised: r e s

Revised: [Figure]

Revised: l

Original: [Figure]
Revised: [Figure]

Original: s t a ge
Revised: s t a g

Original: [Figure]
Revised: [Figure]

Original: r e s b l k 2 s t a g e 4 .

Original: r e s b l k1 s t a g e
Revised: r e s [Figure] l k

Original: [Figure]
Revised: [Figure]

Original: r e s b lk
Revised: f i n a l

Original: [Figure]
Revised: [Figure]

Original: f i n a l _ li n e a r f u ll m o d e l
Revised: li n e a r f u ll [Figure] o d [Figure]

Original: ReRnd
Revised: ReRnd

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  on ImageNet
Revised: (e) ResNet18 on ImageNet

Original: [Figure]
Revised: [Figure]

Original: a smoother transition instead of a sharp contrast at the ﬁrst layer.

Original: We thus measured how distant the weights of each layer from their initialization using the normalized 2-norm and the ∞ -norm, in Figure 1(b) and (c), respectively.

Original: It turns out the robustness to re-initialization does not obviously correlate to either of the distances.

Original: On large ConvNets that are used in practice, similar obversations could be found in Figure 2.
Revised: On large ConvNets that are used in practice, similar obversations could be found in Figure 2.

Original: Moreover, the layerwise robustness patterns for ResNets are quite unique.
Revised: Moreover, the layerwise robustness patterns for ResNets are quite unique.

Original: We found that each stage (in which the “image” size of each layer is the same) in a ResNet acts as a sub-network, and the robustness patterns within each stage resembles the VGGs and FCNs.
Revised: We found that each stage (in which the “image” size of each layer is the same) in a ResNet acts as a sub-network, and the robustness patterns within each stage resembles the VGGs and FCNs.

Original: The skip connections also allow the residual blocks to be robust not only to re-initialization, but also to re-randomization.
Revised: The skip connections also allow the residual blocks to be robust not only to re-initialization, but also to re-randomization.

Original: Please see Appendix B for the full results and analysis.
Revised: Please see Appendix B for the full results and analysis.

Original: To assess the effects of the network capacity and the task complexity on the layer robustness, we apply the same analysis procedure to a large number of different conﬁgurations.
Revised: To assess the effects of the network capacity and the task complexity on the layer robustness, we apply the same analysis procedure to a large number of different configurations.

Original: In Figure 3(a), we compare the average reinitialization robustness for all layers but the ﬁrst with respect FCNs of varying hidden dimensions on MNIST.
Revised: In Figure 3(a), we compare the average re-initialization robustness for all layers but the first with respect FCNs of varying hidden dimensions on MNIST.

Original: The upper layers become more robust as the hidden dimension increases.
Revised: The upper layers become more robust as the hidden dimension increases.

Original: We believe that it reﬂects the fact that the wider FCNs have higher model capacity.
Revised: We believe that it reflects the fact that the wider FCNs have higher model capacity.

Original: When the capacity is small, all layers are vigil participants in representing the prediction function.
Revised: When the capacity is small, all layers are vigil participants in representing the prediction function.

Original: As capacity increases, it sufﬁces to use the bottom layer while the rest act as random projections with non-linearities.
Revised: As capacity increases, it suffices to use the bottom layer while the rest act as random projections with non-linearities.

Original: Similar observations can be found on CIFAR10, in Figure 3(b).
Revised: Similar observations can be found on CIFAR10, in Figure 3(b).

Original: These observations suggest that deep networks automatically adjust their de-facto complexity.
Revised: These observations suggest that deep networks automatically adjust their de-facto complexity.

Original: When a big network is trained on an easy task, only a few layers seem to be playing critical roles.[Figure] [Figure] [Figure]
Revised: When a big network is trained on an easy task, only a few layers seem to be playing critical roles.

Original: A v g

Original: R obu s t ne ss ( i n t e s t

Original: e rr o r)

Original: [Figure]

Original: [Figure]

Original:  Joint robustness and connection to other notions of robustness
Revised:  Joint robustness and connection to other notions of robustness

Original: In all the experiments so far, the analysis focuses on the robustness of each individual layer separately.
Revised: In all the experiments so far, the analysis focuses on the robustness of each individual layer separately.

Original: A natural question is that for all the layers that are individually robustness, are they also jointly robust?
Revised: A natural question is that for all the layers that are individually robustness, are they also jointly robust?

Original: In other words, if we re-initialize multiple layers jointly, would the trained network still retain the original performance?
Revised: In other words, if we re-initialize multiple layers jointly, would the trained network still retain the original performance?

Original: The answer is no.
Revised: The answer is no.

Original: It turns out that jointly re-initialize many consequtive layers (that are individually robust) completely destroy the model performance.
Revised: It turns out that jointly re-initialize many consequtive layers (that are individually robust) completely 512 hidden dimensions 0. 3-layer MLPs 5-layer MLPs A v g

Original: However, via some clever grouping schemes of the layers, multiple layers can be jointly re-initialized while maintaining resonable good model performances.
Revised: R o b u s t n e ss ( i n t e s t e rr o r ) (a) MNIST     512 hidden dimensions 0. 3-layer MLPs 5-layer MLPs (

Revised: [Figure]

Revised: ) CIFAR destroy the model performance.

Revised: However, via some clever grouping schemes of the layers, multiple layers can be jointly re-initialized while maintaining resonable good model performances.

Original: Furthermore, if we explicitly constraing the learning algorithm to not update a subset of layers, much better joint robustness can be achieved.
Revised: Furthermore, if we explicitly constraing the learning algorithm to not update a subset of layers, much better joint robustness can be achieved.

Original: Please see
Revised: Please see

Original: Appendix D for full details.
Revised: Appendix D for full details.

Original: The properties studied in this paper is closely related to a number of other notions of robustness.
Revised: The properties studied in this paper is closely related to a number of other notions of robustness.

Original: For example, the notion of “ﬂatness” refers to the property that neural network weights, after training, can be locally perturbed via isotropic noises without changing the model performance.
Revised: For example, the notion of “flatness” refers to the property that neural network weights, after training, can be locally perturbed via isotropic noises without changing the model performance.

Original: Our study is in a more restricted setting where we perturb by moving along the training trajectory.
Revised: Our study is in a more restricted setting where we perturb by moving along the training trajectory.

Original: More importantly, our empirical results show that while global ﬂatness is quite limited, (a subset of) the individual layers can have negligible affects when perturbed.
Revised: More importantly, our empirical results show that while global flatness is quite limited, (a subset of) the individual layers can have negligible affects when perturbed.

Original: Another interesting connection is to the notion of adversarial robustness to the perturbations applied on the model inputs.
Revised: Another interesting connection is to the notion of adversarial robustness to the perturbations applied on the model inputs.

Original: Please see Appendix E for a full study on those connections.
Revised: Please see Appendix E for a full study on those connections.

Original:  Conclusions
Revised:  Conclusions

Original: We investigated the functional structure on a layer-by-layer basis of over-parameterized deep models.
Revised: We investigated the functional structure on a layer-by-layer basis of over-parameterized deep models.

Original: We introduced the notions of re-initialization and re-randomization robustness.
Revised: We introduced the notions of re-initialization and re-randomization robustness.

Original: Using these notions we provided evidence for the heterogeneous characteristic of layers, which can be morally categorized into either “robust” or “critical”.
Revised: Using these notions we provided evidence for the heterogeneous characteristic of layers, which can be morally categorized into either “robust” or “critical”.

Original: Resetting the robust layers to their initial value has no negative consequence on the model’s performance.
Revised: Resetting the robust layers to their initial value has no negative consequence on the model’s performance.

Original: Our empirical results givefurther evidence thatmere parameter counting or norm accounting is too coarse in studying generalization of deep models. Moreover, optimization landscape based analysis (e.g. ﬂatness or sharpness at the minimizer) is better performed respecting the network architectures due to the heterogeneous behaviors of different layers.
Revised: Our empirical results give evidence that optimization landscape based analysis (e.g. flatness or sharpness at the minimizer) is better performed respecting the network architectures due to the heterogeneous behaviors of different layers.

Original: For future work, we are interested in devising a new algorithm which learns the interleaving trained and partially random subnetworks within one large network.
Revised: For future work, we are interested in devising a new algorithm which learns the interleaving trained and partially random subnetworks within one large network.

Revised: Acknowledgments The authors would like to thank David Grangier, Lechao Xiao, Kunal Talwar and Hanie Sedghi for helpful discussions and comments.

Original: A Related work
Revised: A Related work

Original: Modern neural networks are typically over-parameterized and thus have plenty of redundancy in their representation capabilities.
Revised: Modern neural networks are typically over-parameterized and thus have plenty of redundancy in their representation capabilities.

Original: Previous work exploited over-parameterization to compress [12] or distill [16] a trained network.
Revised: Previous work exploited over-parameterization to compress [12] or distill [16] a trained network.

Original: Rosenfeld and Tsotsos [30] found that one can achieve comparable performance by training only a small fraction of network parameters such as a subset of channels in each convolutional layer.
Revised: Rosenfeld and Tsotsos [30] found that one can achieve comparable performance by training only a small fraction of network parameters such as a subset of channels in each convolutional layer.

Original: Towards interpreting residual networks as ensemble of shallow networks, Veit et al.
Revised: Towards interpreting residual networks as ensemble of shallow networks, Veit et al.

Original: [37] found that residual blocks in a trained network can be deleted or permuted to some extent without hurting the performance too much.
Revised: [37] found that residual blocks in a trained network can be deleted or permuted to some extent without hurting the performance too much.

Original: In another line of research, it is shown that under extreme overparameterization, such as when the network width is polynomial in the training set size and input dimension [1, 6, 7, 42], or even in the asymptotic regime of inﬁnite width [19, 21], the network weights move slowly during training.
Revised: In another line of research, it is shown that under extreme overparameterization, such as when the network width is polynomial in the training set size and input dimension [1, 6, 7, 42], or even in the asymptotic regime of infinite width [19, 21], the network weights move slowly during training.

Original: The observations in this paper show that in more practical regime, different layers could behave very differently.
Revised: The observations in this paper show that in more practical regime, different layers could behave very differently.

Original: B Full results on layerwise robustness analysis of convolutional networks
Revised: B Full results on layerwise robustness analysis of convolutional networks

Original: On typical computer vision tasks beyond MNIST, densely connected FCNs are outperformed signiﬁcantly by convolutional neural networks.
Revised: On typical computer vision tasks beyond MNIST, densely connected FCNs are outperformed significantly by convolutional neural networks.

Original: VGGs and ResNets are among the most widely used convolutional network architectures.
Revised: VGGs and ResNets are among the most widely used convolutional network architectures.

Original: Figure 4 and Figure 5 show the robustness analysis on the two types of networks, respectively.
Revised: Figure 4 and Figure 5 show the robustness analysis on the two types of networks, respectively.

Original: Since those networks are much deeper than the FCNs, we transpose the heatmaps to show the layers as columns.
Revised: Since those networks are much deeper than the FCNs, we transpose the heatmaps to show the layers as columns.

Original: For
Revised: For

Original: VGGs, a large number of layers are sensitive to re-initialization, but the patterns are similar to the observations from the simple FCNs on MNIST: the bottom layers are more critical but the upper layers are robust to re-initialization.
Revised: VGGs, a large number of layers are sensitive to re-initialization, but the patterns are similar to the observations from the simple FCNs on MNIST: the bottom layers are more critical but the upper layers are robust to re-initialization.

Original: The results for ResNets in Figure 5 is to be considered together with results on ImageNet in Figure 6.
Revised: The results for ResNets in Figure 5 is to be considered together with results on ImageNet in Figure 6.

Original: We found the robustness patterns for resnets more interesting mainly for two reasons:
Revised: We found the robustness patterns for resnets more interesting mainly for two reasons:

Original: ResNet s re-distribute sensitive layers.
Revised: ResNet s re-distribute sensitive layers.

Original: Unlike the FCNs and VGGs which put the sensitive layers at the bottom of the network, ResNets distribute them across the network.
Revised: Unlike the FCNs and VGGs which put the sensitive layers at the bottom of the network, ResNets distribute them across the network.

Original: To better understand the patterns, let us do a brief recap of the ResNets architectures.
Revised: To better understand the patterns, let us do a brief recap of the ResNets architectures.

Original: It is common in theoretical analysis to broadly deﬁne ResNets as any neural 
Revised: It is common in theoretical analysis to broadly define ResNets as any neural network architectures with residual blocks . In practice, a few “standard” architectures (and variants) that divide the network into a few “stages” are commonly used. At the bottom, there is a pre-processing stage ( stage0 ) with [Table]

Original: [Table]
Revised: [Table]

Original: ReRnd
Revised: ReRnd

Original: [Table]
Revised: [Table]

Original: r e s b l k

Original: [Figure]
Revised: [Figure]

Original: s t a g e
Revised: [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: r e s b l k
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: s t a g e
Revised: [Table] [Figure] [Table]

Original: [Figure]
Revised: [Figure]

Original: r e s b l k 27 s t a g e
Revised: [Table] [Table] [Table]

Original: [Figure]
Revised: [Table]

Original: r e s b l k 28 s t a g e

Original: [Figure]
Revised: [Table]

Original: r e s b l k 29 s t a g e

Original: [Figure]
Revised: [Table]

Original: r e s b l k 30 s t a g e 
Revised: r e s b l k 25 s t a g e 3 .

Original: [Figure]
Revised: r e s b l k 26 s t a g e 3 .

Original: r e s b l k 31 s t a g e 
Revised: r e s b l k 27 s t a g e 3 .

Original: [Figure]

Original: r e s b l k 32 s t a g e 
Revised: r e s b l k 28 s t a g e 3 .

Original: [Figure]
Revised: r e s b l k 29 s t a g e 3 . r e s b l k 30 s t a g e 3 .

Original: r e s b l k [Figure] s t a g e 
Revised: r e s b l k 31 s t a g e 3 .

Original: [Figure]
Revised: r e s b l k 32 s t a g e 3 .

Original: r e s b l k [Figure] s t a g e 
Revised: r e s b l k 33 s t a g e 3 .

Original: [Figure] r e s [Figure]
Revised: r e s b l k 34 s t a g e 3 .

Original:  l [Figure] s t a g e 
Revised: r e s b l k 35 s t a g e 3 .

Original: [Figure]

Original: r e s b l k 36 s t a g e 4 .
Revised: r e s b l k 36 s t a g e 4 .

Original: r e s b l k 1 s t a g e 4 .
Revised: r e s b l k 1 s t a g e 4 .

Original: r e s b l k 2 s t a g e 4 .
Revised: r e s b l k 2 s t a g e 4 .

Original: r e s b l k 3 f i n a l _ li n e a r f u ll m o d e l
Revised: r e s b l k 3 f i n a l _ li n e a r f u ll m o d e l

Original: [Figure]
Revised: [Figure]

Original: network architectures with residual blocks . In practice, a few “standard” architectures (and variants) that divide the network into a few “stages” are commonly used. At the bottom, there is a pre-processing stage ( stage0 ) with vanilla convolutional layers.
Revised: vanilla convolutional layers.

Original: It is followed by a few (typically 4) residual stages ( stage1 to stage4 ) consisting of residual blocks, and then global average pooling and the densely connected linear classiﬁer ( final_linear ).
Revised: It is followed by a few (typically 4) residual stages ( stage1 to stage4 ) consisting of residual blocks, and then global average pooling and the densely connected linear classifier ( final_linear ).

Original: The image size shrinks and the number of convolutional feature channels doubles from each residual stage to the next one 1 .
Revised: The image size shrinks and the number of convolutional feature channels doubles from each residual stage to the next one 1 .

Original: As a result, while most of the residual blocks have real identity skip connections, the ﬁrst block of each stage ( stage*.resblk1 ) that connects to the previous stage has a non-identity skip connection due to different input / output shapes.
Revised: As a result, while most of the residual blocks have real identity skip connections, the first block of each stage ( stage*.resblk1 ) that connects to the previous stage has a non-identity skip connection due to different input / output shapes.

Original: Figure 7 illustrates the two types of residual blocks.
Revised: Figure 7 illustrates the two types of residual blocks.

Original: With a big picture of the ResNet architectures, we can see that each stage in a ResNet acts as a sub-network, and the layer-wise robustness patterns within each stage resembles the VGGs and FCNs.
Revised: With a big picture of the ResNet architectures, we can see that each stage in a ResNet acts as a sub-network, and the layer-wise robustness patterns within each stage resembles the VGGs and FCNs.

Original: Residual blocks can be robust to re-randomization.
Revised: Residual blocks can be robust to re-randomization.

Original: Among the layers that are robust to re-initialization, if the layer is a residual block, it is also robust to re-randomization: e.g. compare the final_linear layer and any of the robust residual blocks.
Revised: Among the layers that are robust to re-initialization, if the layer is a residual block, it is also robust to re-randomization: e.g. compare the final_linear layer and any of the robust residual blocks.

Original: A possible reason is that the identity skip connection dominates the residual branch in those blocks.
Revised: A possible reason is that the identity skip connection dominates the residual branch in those blocks.

Original: It is known from previous lesion studies [37] that residual blocks in a ResNet can be removed without seriously hurting the performance.
Revised: It is known from previous lesion studies [37] that residual blocks in a ResNet can be removed without seriously hurting the performance.

Original: But our experiments put it in the context with other architectures and study the adaptive robustness with respect to the interplay between the model capacity and the task difﬁculties.
Revised: But our experiments put it in the context with other architectures and study the adaptive robustness with respect to the interplay between the model capacity and the task difficulties.

Original: In particular, comparing the results on CIFAR10 and ImageNet, we see that especially on ResNet18 from Figure 6(a), many residual blocks with real identity skip connection also become sensitive comparing to bigger models due to smaller capacity.
Revised: In particular, comparing the results on CIFAR10 and ImageNet, we see that especially on ResNet18 from Figure 6(a), many residual blocks with real identity skip connection also become sensitive comparing to bigger models due to smaller capacity.

Revised: C Theoretical Implications on Generalization

Revised: As mentioned earlier, if some parameters can be re-assigned to the randomly initialized values without affecting the model performance, then the effective number of parameters is reduced as the random initialization is independent of the training data.

Revised: The benefits on improving generalization is most easily demonstrated with a naive parameter

Original: [Table]
Revised: [Table]

Original: ReRnd
Revised: [Table] [Table] [Figure]

Original: [Table]
Revised: [Table]

Original:  [Table] [Table]
Revised: [Figure] [Table] [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: s b l k 27 s t a g e 3 .
Revised: s b l k 27 s t a g e 3 .

Original: r e s b l k 28 s t a g e 3 .
Revised: r e s b l k 28 s t a g e 3 .

Original: r e s b l k 29 s t a g e 3 . r e s b l k 30 s t a g e 3 .
Revised: r e s b l k 29 s t a g e 3 . r e s b l k 30 s t a g e 3 .

Original: r e s b l k 31 s t a g e 3 .
Revised: r e s b l k 31 s t a g e 3 .

Original: r e s b l k 32 s t a g e 3 .
Revised: r e s b l k 32 s t a g e 3 .

Original: r e s b l k 33 s t a g e 3 . r e s b l k 34 s t a g e 3 .
Revised: r e s b l k 33 s t a g e 3 . r e s b l k 34 s t a g e 3 .

Original: r e s b l k 35 s t a g e 3 .
Revised: r e s b l k 35 s t a g e 3 .

Original: r e s b l k 36 s t a g e 4 .
Revised: r e s b l k 36 s t a g e 4 .

Original: r e s b l k 1 s t a g e 4 .
Revised: r e s b l k 1 s t a g e 4 .

Original: r e s b l k 2 s t a g e 4 .
Revised: r e s b l k 2 s t a g e 4 .

Original: r e s b l k 3 f i n a l _ li n e a r f u ll m o d e l
Revised: r e s b l k 3 f i n a l _ li n e a r f u ll m o d e l

Original: ReRnd1 0.4 0.5 0.
Revised: ReRnd1 0.4 0.5 0.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: As mentioned earlier, if some parameters can be re-assigned to the randomly initialized values without affecting the model performance, then the effective number of parameters is reduced as the random initialization is independent of the training data. The beneﬁts on improving generalization is most easily demonstrated with a naive parameter counting generalization bound.
Revised: counting generalization bound.

Original: For example, if we have a generalization bound of the form
Revised: For example, if we have a generalization bound of the form

Original: [Equation]
Revised: [Equation]

Original: where ˆ f mn is a model with m parameters trained on n i.i.d.
Revised: where ̂𝑓 𝑚𝑛 is a model with 𝑚 parameters trained on 𝑛 i.i.d.

Original: samples.
Revised: samples.

Original: C ( · ) is some complexity measure based on counting the number of parameters, and B is the corresponding generalization bound.
Revised:  (⋅) is some complexity measure based on counting the number of parameters, and  is the corresponding generalization bound.

Original: For example, Anthony and Bartlett [2] provides various bounds on VC-dimension based on the number of weights in neural networks, which could then be plugged into standard VC-dimension based generalization bounds for classiﬁcation [36].
Revised: For example, Anthony and Bartlett [2] provides various bounds on VC-dimension based on the number of weights in neural networks, which could then be plugged into standard VC-dimension based generalization bounds for classification [36].

Original: Now if we know that a fraction ρ ∈ (0,1) of the neural network weights will be robust to re-initialization after training, with a[Figure] loss of the (empirical) risk of at most ε , then we get
Revised: Now if we know that a fraction 𝜌 ∈ (0, 1) of the neural network weights will be robust to re-initialization after training, with a loss of the (empirical) risk of at most 𝜀 , then we get

Original: [Equation]
Revised: [Equation]

Original: where ˆ f (1 − ρ ) m n is a model obtained by re-initializing the ρ fraction of parameters of the trained model ˆ f mn .
Revised: where ̂𝑓 (1−𝜌)𝑚𝑛 is a model obtained by re-initializing the 𝜌 fraction of parameters of the trained model ̂𝑓 𝑚𝑛 .

Original: Note that generalization bounds based on parameter counting generally does not work well for deep learning.
Revised: Note that generalization bounds based on parameter counting generally does not work well for deep learning.

Original: Because of the  heavy over-parameterization, the resulting bounds are usually trivial.
Revised: Because of the [Figure] heavy over-parameterization, the resulting bounds are usually trivial.

Original: However, as noted in Arora et al.
Revised: However, as noted in Arora et al.

Original: [3], most of the alternative generalization bounds proposed for deep neural network models recently are actually worse than naive parameter counting.
Revised: [3], most of the alternative generalization bounds proposed for deep neural network models recently are actually worse than naive parameter counting.

Original: Moreover, by tweaking the existing analysis with additional layerwise robustness condition, some PAC-Bayes based bounds can also be potentially improved [38, 3, 41].
Revised: Moreover, by tweaking the existing analysis with additional layerwise robustness condition, some PAC-Bayes based bounds can also be potentially improved [38, 3, 41].

Original: Note that like the results in Arora et al.
Revised: Note that like the results in Arora et al.

Original: [3], Zhou et al.
Revised: [3], Zhou et al.

Original: [41], the bounds provided by re-initialization robustness are for a different model (in our case the re-initialized one).
Revised: [41], the bounds provided by re-initialization robustness are for a different model (in our case the re-initialized one).

Original: Alternative approaches in the literature involve modifying the training algorithms to explicitly optimize the robustness or some derived generalization bounds [25, 8].
Revised: Alternative approaches in the literature involve modifying the training algorithms to explicitly optimize the robustness or some derived generalization bounds [25, 8].

Original: However, neither of the arguments provides guarantees for the model directly trained from SGD.
Revised: However, neither of the arguments provides guarantees for the model directly trained from SGD.

Original: D Joint robustness
Revised: D Joint robustness

Original: The theoretical analysis suggests that robustness to either re-initialization or re-randomization could imply better generalization.
Revised: The theoretical analysis suggests that robustness to either re-initialization or re-randomization could imply better generalization.

Original: Combined with the experimental results in previous sections, it seems to suggest a good way to explain the empirical observations that hugely over-parameterized networks could still generalize well, as they are only using a small portion of their full capacity.
Revised: Combined with the experimental results in previous sections, it seems to suggest a good way to explain the empirical observations that hugely over-parameterized networks could still generalize well, as they are only using a small portion of their full capacity.

Original: However, there is a caveat: the re-initialization and re-randomization analysis in Section 3 study each layer independently.
Revised: However, there is a caveat: the re-initialization and re-randomization analysis in Section 3 study each layer independently.

Original: However, two or more layers being independently robust does not necessarily imply that they are robust jointly .
Revised: However, two or more layers being independently robust does not necessarily imply that they are robust jointly .

Original: If, for example, we want a generalization bound that uses only half of the capacity, we need to show that half of the layers are robust to re-initialization or re-randomization simultaneously .
Revised: If, for example, we want a generalization bound that uses only half of the capacity, we need to show that half of the layers are robust to re-initialization or re-randomization simultaneously .

Original: D. Are robust layers jointly robust?
Revised: D. Are robust layers jointly robust?

Original: In this section, we do joint robustness analysis on groups of layers.
Revised: In this section, we do joint robustness analysis on groups of layers.

Original: From Section 3, we see that on MNIST, for wide enough FCNs, all the layers above layer1 are robust to re-initialization.
Revised: From Section 3, we see that on MNIST, for wide enough FCNs, all the layers above layer1 are robust to re-initialization.

Original: So we divide the layer into two groups: { layer1 } and { layer2 , layer3 , ...}, and perform the robustness studies on the two groups.
Revised: So we divide the layer into two groups: { layer1 } and { layer2 , layer3 , ...}, and perform the robustness studies on the two groups.

Original: The results for FCN 5 × 256 are shown in Figure 8(a).
Revised: The results for FCN 5 ×are shown in Figure 8(a).

Original: For clarity and ease of comparison, the ﬁgure still spells out all the layers individually, but the values from layer2 to layer6 are simply repeated rows.
Revised: For clarity and ease of comparison, the figure still spells out all the layers individually, but the values from layer2 to layer6 are simply repeated rows.

Original: The values show that the upper-layer-group is clearly not jointly robust to re-initialization (to checkpoint 0).[Table] [Figure] [Table]
Revised: The values show that the upper-layer-group is clearly not jointly robust to re-initialization (to checkpoint 0).

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: s t a g e

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Table]

Original: ReRnd

Original: [Table]

Original: [Table]

Original: [Table]

Original: l k 24 * s t a g e 3 .

Original: r e s b l k 25 s t a g e 3 .

Original: r e s b l k 26 * s t a g e 3 .

Original: r e s b l k 27 s t a g e 3 .

Original: r e s b l k 28 * s t a g e 3 .

Original: r e s b l k 29 s t a g e 3 .

Original: r e s b l k 30 * s t a g e 3 .

Original: r e s b l k 31 s t a g e 3 .

Original: r e s b l k 32 * s t a g e 3 .

Original: r e s b l k 33 s t a g e 3 .

Original: r e s b l k 34 * s t a g e 3 .

Original: r e s b l k 35 s t a g e 3 .

Original: r e s b l k 36 * s t a g e 4 .

Original: r e s b l k 1 s t a g e 4 .

Original: r e s b l k 2 * s t a g e 4 .

Original: r e s b l k 3 f i n a l _ li n e a r

Original: ReRnd1 0.1 0.2 0. [Figure] [Table] We also try some alternative grouping schemes: Figure 8(b) show the results when we group two in every three layers
Revised: We also try some alternative grouping schemes: Figure 8(b) show the results when we group two in every three layers, which has slightly improved joint robustness;

Original: [Table]

Original: which has slightly improved joint robustness; In Figure 8(c), the grouping scheme that include every other layer shows that with a clever grouping scheme, about half of the layers could be jointly robust.
Revised: In Figure 8(c), the grouping scheme that include every other layer shows that with a clever grouping scheme, about half of the layers could be jointly robust.

Original: Results on ResNets are similar.
Revised: Results on ResNets are similar.

Original: Figure 9 shows the joint robustness analysis on ResNets trained on CIFAR10.
Revised: Figure 9 shows the joint robustness analysis on ResNets trained on CIFAR10.

Original: The grouping is based on the layer-wise robustness results from Figure 5: all the residual blocks in stage1 to stage4 are bundled and analyzed jointly.
Revised: The grouping is based on the layer-wise robustness results from Figure 5: all the residual blocks in stage1 to stage4 are bundled and analyzed jointly.

Original: The results are similar to the FCNs: ResNet18 is relatively robust, but deeper ResNets are not jointly robust under this grouping.
Revised: The results are similar to the FCNs: ResNet18 is relatively robust, but deeper ResNets are not jointly robust under this grouping.

Original: Two alternative grouping schemes are shown in Figure 10.
Revised: Two alternative grouping schemes are shown in Figure 10.

Original: By including only layers from stage1 and stage4 , slightly improved robustness could be obtained on ResNet50.
Revised: By including [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Revised: s t a g e

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Table]

Revised: ReRnd

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: only layers from stage1 and stage4 , slightly improved robustness could be obtained on ResNet50.

Original: The scheme that groups every other residual block shows further improvements.
Revised: The scheme that groups every other residual block shows further improvements.

Original: In summary, the individually robust layers are generally not jointly robust.
Revised: In summary [Table] the individually robust layers are generally not jointly robust.

Original: But with some clever way of picking out a subset of the layers, joint robustness could still be achieved for up to half of the layers.
Revised: But with some clever way of picking out a subset of the layers, joint robustness could still be achieved for up to half of the layers.

Original: In principle, one can enumerate all possible grouping schemes to ﬁnd the best with a trade-off of the robustness and number of layers included.
Revised: In principle, one can enumerate all possible grouping schemes to find the best with a trade-off of the robustness and number of layers included.

Original: D. Could robust layers be made jointly robust?
Revised: D. Could robust layers be made jointly robust?

Original: Results from the previous section show that there is a gap between the layer-wise robustness patterns and the the joint robustness.
Revised: Results from the previous section show that there is a gap between the layer-wise robustness patterns and the the joint robustness.

Original: Here we try to see if we could close the gap by letting the training algorithm know that we are interested in the robustness of a subset of the layers.
Revised: Here we try to see if we could close the gap by letting the training algorithm know that we are interested in the robustness of a subset of the layers.

Original: It is complicated to express this desire algorithmically, but we can make a stronger request by asking the learning algorithm to explicitly not “use” those layers.
Revised: It is complicated to express this desire algorithmically, but we can make a stronger request by asking the learning algorithm to explicitly not “use” those layers.

Original: More speciﬁcally, we try two approaches to the layers in the group that is desired to be robust: 1) freeze them so that their parameters remain the same randomly initialized values; 2) remove the layers completely from the neural network architecture.
Revised: More specifically, we try two approaches to the layers in the group that is desired to be robust: 1) freeze them so that their parameters remain the same randomly initialized values; 2) remove the layers completely from the neural network architecture.

Original: The results are shown in Table 1.
Revised: The results are shown in Table 1.

Original: When we explicitly freeze the layers, the test error rates are still higher than the average layer-wise robustness measured in a normally trained model.
Revised: When we explicitly freeze the layers, the test error rates are still higher than the average layer-wise robustness measured in a normally trained model.

Original: However, the gap is much smaller than directly measuring the joint robustness (see Figure 9 for comparison).
Revised: However, the gap is much smaller than directly measuring the joint robustness (see Figure 9 for comparison).

Original: Moreover, on CIFAR10, we found that similar performance can be achieved even if we completely remove those layers from the network.
Revised: Moreover, on CIFAR10, we found that similar performance can be achieved even if we completely remove those layers from the network.

Original: On the other hand, on
Revised: On the other hand, on

Original: ImageNet, the frozen random layers seem to be needed to achieve good performances, while the “layers-removed” variant under-perform by a big gap.
Revised: ImageNet, the frozen random layers seem to be needed to achieve good performances, while the “layers-removed” variant under-perform by a big gap.

Original: In this case, the random projections (with non-linearity) in those frozen layers are helpful with the performance.[Table]
Revised: In this case, the random projections (with non-linearity) in those frozen layers are helpful with the performance.

Original: E Connections to other notions of robustness
Revised: E Connections to other notions of robustness

Original: The notion of layer-wise (and joint) robustness to re-initialization and re-randomization can be related to other notions of robustness in deep learning.
Revised: The notion of layer-wise (and joint) robustness to re-initialization and re-randomization can be related to other notions of robustness in deep learning.

Original: For example, the ﬂatness of the solution is a notion of robustness with respect to local perturbations to the network parameters (at convergence), and is extensively discussed in the context of generalization [17, 4, 20, 33, 28].
Revised: For example, the flatness of the solution is a notion of robustness with respect to local perturbations to the network parameters (at convergence), and is extensively discussed in the context of generalization [17, 4, 20, 33, 28].

Original: For a ﬁxed layer, our notion of robustness to re-initialization is more restricted because the “perturbed values” can only be from the optimization trajectory; while the robustness to re-randomization could potentially allow larger perturbation variances.
Revised: For a fixed layer, our notion of robustness to re-initialization is more restricted because the “perturbed values” can only be from the optimization trajectory; while the robustness to re-randomization could potentially allow larger perturbation variances.

Original: However, as our studies here show, the robustness or ﬂatness at each layer could behave very differently, so analyzing each layer individually in the context of speciﬁc network architectures allow us to get more insights to the robustness behaviors.
Revised: However, as our studies here show, the robustness or flatness at each layer could behave very differently, so analyzing each layer individually in the context of specific network architectures allow us to get more insights to the robustness behaviors.

Original: On the other hand, Adversarial robustness [34] focus on the robustness with respect to perturbations to the inputs. 
Revised: On the other hand, Adversarial robustness [34] focus on the robustness with respect to perturbations to the inputs. In

Original: In particular, it is found that trained deep neural network models are sensitive to input perturbations: small adversarially generated perturbations can usually change the prediction results to arbitrary different classes.
Revised: [Table] particular [Table] it is found that trained deep neural network models are sensitive to input perturbations: small adversarially generated perturbations can usually change the prediction results to arbitrary different classes.

Original: A large number of defending and attacking algorithms have been proposed in recent years along this line.
Revised: A large number of defending and attacking algorithms have been proposed in recent years along this line.

Original: Here we brieﬂy discuss the connection to adversarial robustness.
Revised: Here we briefly discuss the connection to adversarial robustness.

Original: In particular, take a normally trained ResNet 2 , say with S stages and ( B 1 ,..., B S ) residual blocks in each stage.
Revised: In particular, take a normally trained ResNet 2 , say with 𝑆 stages and (𝐵 1 , … , 𝐵 𝑆 ) residual blocks in each stage.

Original: Given conﬁguration r > 0 and 0 ≤ s ≤ S , during each test evaluation, a subset of s stages are randomly chosen, and for each of the chosen stages, a random residual block is picked and replaced with one of the r pre-initialized weights for that layer.
Revised: Given configuration 𝑟 > 0 and 0 ≤ 𝑠 ≤ 𝑆 , during each test evaluation, a subset of 𝑠 stages are randomly chosen, and for each of the chosen stages, a random residual block is picked and replaced with one of the 𝑟 pre-initialized weights for that layer.

Original: We keep r pre-allocated weights for each residual block instead of re-sampling random numbers on each evaluation call, primarily to reduce the computation burden during the test time.
Revised: We keep 𝑟 pre-allocated weights for each residual block instead of re-sampling random numbers on each evaluation call, primarily to reduce the computation burden during the test time.

Original: From the previous robustness analysis, we expect the stochastic classiﬁer to get only a small performance drop when averaged over the test set.
Revised: From the previous robustness analysis, we expect the stochastic classifier to get only a small performance drop when averaged over the test set.

Original: However, at individual example level, the randomness of the network outputs will make it harder for the attacker to generate adversarial examples.
Revised: However, at individual example level, the randomness of the network outputs will make it harder for the attacker to generate adversarial examples.

Original: We evaluate the adversarial robustness against a weak FGSM [10] attack and a strong PGD [22] attack.
Revised: We evaluate the adversarial robustness against a weak FGSM [10] attack and a strong PGD [22] attack.

Original: The results in Table 2 show that, compared to the baseline (the exact same trained model before being turned into a stochastic classiﬁer), the randomness signiﬁcantly increases the adversarial robustness against weak attacks.
Revised: The results in Table 2 show that, compared to the baseline (the exact same trained model before being turned into a stochastic classifier), the randomness significantly increases the adversarial robustness against weak attacks.

Original: The performances under strong PGD attack drop to very low, but still with a non-trivial gap between the baseline.
Revised: The performances under strong PGD attack drop to very low, but still with a non-trivial gap between the baseline.

Original: In summary, the layer-wise robustness could improve the adversarial robustness of a trained model through injected stochasticity.
Revised: In summary, the layer-wise robustness could improve the adversarial robustness of a trained model through injected stochasticity.

Original: However, it is not a good defense against strong attackers.
Revised: However, it is not a good defense against strong attackers.

Original: If we work hard enough, more sophisticated attacks that explicitly deal with stochastic classiﬁers are likely to completely break this model.
Revised: If we work hard enough, more sophisticated attacks that explicitly deal with stochastic classifiers are likely to completely break this model.

Original: F Details on experiment setup
Revised: F Details on experiment setup

Original: Our empirical studies are based on the MNIST, CIFAR10 and the ILSVRC 2012 ImageNet datasets.
Revised: Our empirical studies are based on the MNIST, CIFAR10 and the ILSVRC 2012 ImageNet datasets.

Original: Stochastic 
Revised: Stochastic Gradient

Original: Gradient Descent (SGD) with a momentum of 0.9 is used to minimize the multi-class cross entropy loss.
Revised: Descent (SGD) with a momentum of 0.9 is used to minimize the multi-class cross entropy loss.

Original: Each model is trained for 100 epochs, using a stage-wise constant learning rate scheduling with a multiplicative factor of 0.2 on epoch 30, 60 and 90.
Revised: Each model is trained for 100 epochs, using a stage-wise constant learning rate scheduling with a multiplicative factor of 0.2 on epoch 30, 60 and 90.

Original: Batch size of 128 is used, except for ResNets with more than 50 layers on ImageNet, where batch size of 64 is used due to device memory constraints.
Revised: Batch size of 128 is used, except for ResNets with more than 50 layers on ImageNet, where batch size ofis used due to device memory constraints.

Original: We mainly study three types of neural network architectures:
Revised: We mainly study three types of neural network architectures:

Original: • FCNs: the multi-layer perceptrons consist of fully connected layers with equal output dimension and ReLU activation (except for the last layer, where the output dimension equals the number of classes and no ReLU is  applied). 
Revised: • FCNs: the multi-layer perceptrons consist of fully connected layers with equal output dimension and ReLU activation (except for the last layer, where the output dimension equals the number of classes and no ReLU is [Table] applied). For example

Revised: [Table]

Original: For example, FCN 3 × 256 has three layers of fully connected layers with the output dimension 256, and an extra ﬁnal (fully connected) classiﬁer layer.
Revised: FCN 3 × 256 has three layers of fully connected layers with the output dimension , and an extra final (fully connected) classifier layer.

Original: • VGGs: widely used network architectures from Simonyan and Zisserman [32].
Revised: • VGGs: widely used network architectures from Simonyan and Zisserman [32].

Original: • ResNets: the results from our analysis are similar for ResNets V1 [14] and V2 [15].
Revised: • ResNets: the results from our analysis are similar for ResNets V1 [14] and V2 [15].

Original: We report our results with ResNets V2 due to the slightly better performance in most of the cases.
Revised: We report our results with ResNets V2 due to the slightly better performance in most of the cases.

Original: For large image sizes from ImageNet, the stage0 contains a 7 × 7 convolution and a 3 × 3 max pooling (both with stride 2) to reduce the spatial dimension (from 224 to56).
Revised: For large image sizes from ImageNet, the stage0 contains a 7 × 7 convolution and a 3 × 3 max pooling (both with stride 2) to reduce the spatial dimension (from 224 to

Revised: On

Original: On smaller image sizes like CIFAR10, we use a 3 × 3 convolution with stride 1 here to avoid reducing the spatial dimension.
Revised: smaller image sizes like CIFAR , we use a 3 × 3 convolution with stride 1 here to avoid reducing the spatial dimension

Original: During training, CIFAR10 images are padded with 4 pixels of zeros on all sides, then randomly ﬂipped (horizontally) and cropped.
Revised: During training, CIFAR10 images are padded with 4 pixels of zeros on all sides, then randomly flipped (horizontally) and cropped.

Original: ImageNet images are randomly cropped during training and center-cropped during testing.
Revised: ImageNet images are randomly cropped during training and center-cropped during testing.

Original: Global mean and standard deviation are computed on all the training pixels and applied to normalize the inputs on each dataset.
Revised: Global mean and standard deviation are computed on all the training pixels and applied to normalize the inputs on each dataset.

Original: G Batch normalization and weight decay
Revised: G Batch normalization and weight decay

Original: The primary goal of this paper is to study the (co-)evolution of the representations at each layer during training and the robustness of this representation with respect to the rest of the network.
Revised: The primary goal of this paper is to study the (co-)evolution of the representations at each layer during training and the robustness of this representation with respect to the rest of the network.

Original: We try to minimize the factors that explicitly encourage changing of the network weights or representations in the analysis.
Revised: We try to minimize the factors that explicitly encourage changing of the network weights or representations in the analysis.

Original: In particular, unless otherwise speciﬁed, weight decay and batch normalization are not used.
Revised: In particular, unless otherwise specified, weight decay and batch normalization are not used.

Original: This leads to some performance drop in the trained models.
Revised: This leads to some performance drop in the trained models.

Original: Especially for deep residual networks: even though we could successfully train a residual network with 100+ layers without batch normalization, the ﬁnal generalization performance could be quite worse than the state-of-the-art.
Revised: Especially for deep residual networks: even though we could successfully train a residual network with 100+ layers without batch normalization, the final generalization performance could be quite worse than the state-of-the-art.

Original: Therefore, in this section, we include studies on networks trained with weight decay and batch normalization for comparison.
Revised: Therefore, in this section, we include studies on networks trained with weight decay and batch normalization for comparison.

Original: In particular, Table 3 shows the ﬁnal test error rates of models trained with or without weight decay and batch normalization.
Revised: In particular, Table 3 shows the final test error rates of models trained with or without weight decay and batch normalization.

Original: Note the original VGG models do not use batch normalization [32], we list +bn variants here for comparison, by applying batch normalization to the output of each convolutional layer.
Revised: Note the original VGG models do not use batch normalization [32], we list +bn variants here for comparison, by applying batch normalization to the output of each convolutional layer.

Original: On CIFAR10, the performance gap varies from 3% to 5%, but on ImageNet, a performance gap as large as 10% could be seen when trained without weight decay and batch normalization.
Revised: On CIFAR10, the performance gap varies from 3% to 5%, but on ImageNet, a performance gap as large as 10% could be seen when trained without weight decay and batch normalization.

Original: Figure 11 shows how different training conﬁgurations affect the layerwise robustness analysis patterns on VGG16 networks.
Revised: Figure 11 shows how different training configurations affect the layerwise robustness analysis patterns on VGG16 networks.

Original: We found that when batch normalization is used, none of the layersare robust any more.
Revised: We found that when batch normalization is used, none of the layers

Original: Figure 12 and Figure 13 show similar comparisons for ResNet50 on CIFAR10 and ImageNet, respectively.
Revised: [Table] [Table] [Table]

Original: Unlike VGGs, we found that the layerwise robustness patterns are still quite pronounced under various training conditions for
Revised: [Table]

Original: ResNets.
Revised: [Table]

Original: In Figure 12(d) and Figure 13(c,d), we see the mysterious phenomenon that re-initialing with checkpointis less robust than with checkpoint-0 for many layers.
Revised: [Table]

Original: We do not know exactly why this is happening.
Revised: [Table] [Table] [Figure]

Original: It might be that during early stages, some aggressive learning is happening causing changes in the parameters or statistics with large magnitudes, but later on when most of the training samples are classiﬁed correctly, the network gradually rebalances the layers to a more robust state. Figure 15(d-f) in the next section shows supportive evidence that, in this case the distance of the parameters between checkpoint-0 and checkpoint-1 is larger than between checkpoint-
Revised: bn) affects the layerwise robustness patterns

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Revised: ReRnd

Original: [Table] [Table]
Revised: [Table] [Table]

Original: s t a g e 5 . b l o c k 3 f c 1 f c 2 f i n a l _ li n e a r f u ll m o d e l
Revised: [Table] [Table]

Original: ReRnd0.1 0.2 0.
Revised: ReRnd

Original: [Figure]
Revised: [Table]

Original: VGG

Original: [Figure]
Revised: [Table]

Original: s t a g e 1 .  b lo c k 1 s t a g e 1 .
Revised: a g e 3 . r e s b l k 1 s t a g e 3 .

Original:  b lo c k 2 s t a g e 2 . b l o c k 1 s t a g e 2 . b lo c k 2 s t a g e 3 .
Revised: r e s b l k 2 s t a g e 3 . r e s b l k 3 s t a g e 3 .

Original: b l o c k 1 s t a g e3 . b l o c k 2 st a g e 3 . b lo c k3 s t a g e 4. b l o c k 1 s t a g e 4 .
Revised: r e s b l k 4 s t a g e 3 .

Original:  b lo c k 2 s t a g e 4 .  b lo c k 3 s t a g e 5 .
Revised: r e s b l k 5 s t a g e 3 . r e s b l k 6 s t a g e 4 .

Original:  b lo c k 1 s t a g e 5 .
Revised: r e s b l k 1 s t a g e 4 .

Original:  b lo c k 2 s t a g e 5 .
Revised: r e s b l k 2 s t a g e 4 .

Original:  b lo c k 3 fc 1 f c 2 f i n a l _ li n e a r f u ll m o d e l
Revised: r e s b l k 3 f i n a l _ li n e a r f u ll m o d e l

Original: ReRnd    0.1 0.2 0.
Revised: ReRnd    0.1 0.2 0.

Original: [Figure]
Revised: [Figure]

Revised: are robust any more.

Revised: Figure 12 and Figure 13 show similar comparisons for ResNet50 on CIFAR10 and ImageNet, respectively.

Revised: Unlike VGGs, we found that the layerwise robustness patterns are still quite pronounced under various training conditions for

Revised: ResNets.

Original:  and the ﬁnal checkpoint.
Revised: In Figure 12(d) and Figure 13(c,d), we see the mysterious phenomenon that re-initialing with checkpoint-1 is less robust than with checkpoint-0 for many layers. We do not know exactly why this is happening. It might be that during early stages, some aggressive learning is happening causing changes in the parameters or statistics with large magnitudes, but later on when most of the training samples are classified correctly, the network gradually re-balances the layers to a more robust state. Figure 15(d-f) in the next section shows supportive evidence that, in this case the distance of the parameters between checkpoint-0 and checkpoint-1 is larger than between checkpoint-0 and the final checkpoint.

Original: However, on ImageNet this correlation is no longer clear as seen in Figure 16(d-
Revised: However, on ImageNet this correlation is no longer clear as seen in Figure 16(d-f).

Original: See the discussions in the next section for  details.
Revised: See the discussions in the next section for more details.

Original: H Robustness and distances
Revised: [Table] ReRnd

Original: In Figure 1 from Section 3, we compared the layerwise robustness pattern to the layerwise distances of the parameters to the values at initialization (checkpoint-0). We found that for FCNs on MNIST, there is no obvious correlation between the “amount of parameter updates received” at each layer and its robustness
Revised: [Table] [Figure]

Original: [Table]
Revised: [Figure]

Original: n a l

Original: [Table] n e a r [Table] u
Revised: H Robustness and distances

Original: [Table]

Original: m [Table] [Figure] [Figure]
Revised: In Figure 1 from Section

Original: ReRnd

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Table]

Original: [Table] [Table] [Table] [Table]
Revised: we compared the layerwise robustness pattern to the layerwise distances of the parameters to the values at initialization (checkpoint-0).

Original: t a g e 4 . b l o c k 2 s t a g e 4 . b l o c k 3 s t a g e 5 . b l o c k 1 s t a g e 5 .
Revised: We found that for FCNs on MNIST, there is no obvious correlation between the “amount of parameter updates received” at each layer and its robustness to re-initialization for the two distances (the normalized 2 and ∞ norms) we measured.

Original: b l o c k 2 s t a g e 5 . b l o c k 3 f c 1 f c 2 f i n a l _ li n e a r [Figure] [Table]
Revised: In this appendix, we list results on other models and datasets studied in this paper for comparison.

Original: [Figure]

Original: [Figure]

Original: [-ReRnd
 [Table] [Figure]-] Figure 14 shows the layerwise robustness plot along with the layerwise distance plots for VGG16 trained on CIFAR10.
Revised: [-ReRnd
 [Table] [Figure]-] Figure 14 shows the layerwise robustness plot along with the layerwise distance plots for VGG16 trained on CIFAR10.

Original: We found that the (cid:96) ∞ distance of the top layers are large, but the model is robust when we re-initialize those layers.
Revised: We found that the 𝓁 ∞ distance of the top layers are large, but the model is robust when we re-initialize those layers.

Original: However, the normalized (cid:96) 2 distance seem to be correlated with the layerwise robustness patterns: the lower layers that are less robust have larger distances to their initialized values.
Revised: However, the normalized 𝓁 2 distance seem to be correlated with the layerwise robustness patterns: the lower layers that are less robust have larger distances to their initialized values.

Original: Similar plots for ResNet50 on CIFAR10 and ImageNet are shown in Figure 15 and Figure 16, respectively.
Revised: Similar plots for ResNet50 on CIFAR10 and ImageNet are shown in Figure 15 and Figure 16, respectively.

Original: In each of the ﬁgures, we also show extra results for models trained with weight decay and batch normalization.
Revised: In each of the figures, we also show extra results for models trained with weight decay and batch normalization.

Original: For the case without weight decay and batch normalization, we can see a weak correlation: the layers that are sensitive have slightly larger distances to their random initialization values.
Revised: For the case without weight decay and batch normalization, we can see a weak correlation: the layers that are sensitive have slightly larger distances to their random initialization values.

Original: For the case with weight decay and batch normalization, the situation is less clear.
Revised: For the case with weight decay and batch normalization, the situation is less clear.

Original: First of all, in Figure 15(e-f), we see very large distances in a few layers at checkpoint-1.
Revised: First of all, in Figure 15(e-f), we see very large distances in a few layers at checkpoint-1.

Original: This provides a potential explanation to the mysterious pattern that re-initialization to checkpointis more sensitive than to checkpoint-0.
Revised: This provides a potential explanation to the mysterious pattern that re-initialization to checkpoint-1 is more sensitive than to checkpoint-0.

Original: Similar observations can be found in Figure 16(e-f) for ImageNet. 
Revised: Similar observations can be found in Figure 16(e-f) for ImageNet. [Table] [Figure] [Table]

Original: I Alternative visualizations
Revised: [Figure] [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: {+ReRnd
+}
Revised: {+ReRnd
+}

Revised: [Figure]

Original: The empirical results on layer robustness are mainly visualized as heatmaps in the main text.
Revised: [Figure] wd [Figure] bn

Original: The heatmaps allow uncluttered comparison of the results across layers and training epochs.
Revised: [Figure]

Original: However, it is not easy to tell the difference between numerical values that are close to each other from the color coding.
Revised: g

Original: In this section, we provide alternative s t a g e 1 . b l o c k 1 s t a g e 1 . b l o c k 2 s t a g e 2 . b l o c k 1 s t a
Revised: [Table]

Original: [Table] e 2 . b l o [Table]
Revised: [Figure]

Original: k 2 s t a

Original: [Table]

Original: e 3 .

Original: b l o c k

Original: [Table]

Original: s t a

Original: [Table]

Original: e 3 . [Table] l o [Table]
Revised: I Alternative visualizations

Original: k 2 s t a

Original: [Table]

Original: e 3 .

Original: b l o c k 3 s t a g e 4 . b l o c k 1 s t a g e 4 . b l o c k 2 s t a g e 4 . b l o c k 3 s t a g e 5 .
Revised: The empirical results on layer robustness are mainly visualized as heatmaps in the main text.

Original: b l o c k 1 s t a g e 5 .

Original: b l o c k 2 s t a g e 5 .

Original: b l o c k 3 f c 1 f c 2 f i n a l _ li n e a r 0.2 0.

Original: final epoch ReRnd ReInit to E0 ReInit to E ReInit to E (a [Table]
Revised: The heatmaps allow uncluttered comparison of the results across layers and training epochs.

Original: s t a

Original: [Table]

Original: e 1 .

Original: b l o c k 1 s t a [Table] e 1 . b l o c k 2 s t a g e 2 .
Revised: However, it is not easy to tell the difference between numerical values that are close to each other from the color coding.

Original: b l o c k 1 s t a g e 2 .

Original: b l o c k 2 s t a g e 3 .

Original: b l o c k 1 s t a g e 3 . b l o c k 2 s t a g e 3 . b l o c k 3 s t a g e 4 . b l o c k 1 s t a g e 4 .
Revised: In this section, we provide alternative visualizations that shows the same results with line plots.

Original: b l o c k 2 s t a g e 4 . b l o c k 3 s t a g e 5 . b l o c k 1 s t a g e 5 . b l o c k 2 s t a g e 5 .
Revised: In particular, Figure 17 shows the layerwise robustness analysis for VGG16 on CIFAR10.

Original: b l o c k 3 f c 1 f c 2 f i n a l _ li n e a r 0. final epoch ReRnd ReInit to E0 ReInit to E ReInit to E (
Revised: Figure 18 and Figure 19 show the results for ResNet50 on CIFAR10 and ImageNet, respectively.

Original: [Table]
Revised: [Table]

Original: s t a g e 1 .

Original: b l o c k 1 s t a g e 1 .

Original: b l o c k

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: e

Original: [Figure]

Original: t

Original: [Figure]

Original: g

Original: [Figure]
Revised: [Figure]

Original: r e s b l k 6 s t a g e 4 .

Original: r e s b l k 1 s t a g e 4 .

Original: r e s b l k 2 s t a g e 4 .

Original: r e s b l k

Original: [Figure]
Revised: [Figure]

Original: f i n a l _ li n e a r

Original: [Figure] visualizations that shows the same results with line plots. In particular, Figure 17 shows the layerwise robustness analysis for VGG16 on CIFAR10. Figure 18 and Figure 19 show the results for ResNet50 on CIFAR10 and ImageNet, respectively.
Revised: [Table]

Original: [Figure][Table] [Figure]
Revised: [Figure] [Figure]