Understanding learning and generalization of deep architectures has been a major research objective in the recent years with notable theoretical progress. A main focal point of generalization studies stems from the success of excessively large networks which defy the classical wisdom of uniform convergence and learnability. We study empirically the layer-wise functional structure of overparameterized deep models. We provide evidence for the heterogeneous characteristic of layers. To do so, we introduce the notion of (post training) re-initialization and re-randomization robustness. We show that layers can be categorized into either "robust" or "critical". In contrast to critical layers, resetting the robust layers to their initial value has no negative consequence, and in many cases they barely change throughout training. Our study provides further evidence that mere parameter counting or norm accounting is too coarse in studying generalization of deep models, and flatness or robustness analysis of the model parameters needs to respect the network architectures. After training, for a given layer d = 1, . . . , D, we can re-initialize the parameters through assignment θ T d ← θ 0 d , while keeping the parameters for the other layers unchanged. The model with the parameters (θ T 1 , . . . , is then evaluated. Unless noted otherwise, we use the term performance to designate classification error on test data. The performance of a network in which layer d was re-initialized is referred to as the re-initialization robustness of layer d . Note that here θ 0 d denotes the random values realized at the beginning of the training. More generally, for k time steps 0 = τ 1 < τ 2 < · · · < τ k−1 < τ k = T , we can re-initialize the d -th layer by setting θ T d ← θ τ d , and obtain the re-initialization robustness of layer d after τ updates. To go one step further, we also examine re-randomization of a layer d by re-sampling random valuesθ d ∼ P d and evaluate the model's performance for (θ T 1 , . . . , θ T d −1 ,θ d , θ T d +1 , . . . , θ T D ). Analogously, we refer to the evaluated performance as the re-randomization robustness of layer d . Note that there is no re-training or finetuning after re-initialization or re-randomization, and the network is evaluated directly with mixed weights. When a network exhibits no or negligible decrease in performance after re-initializing or re-randomizing of a layer, we say that the layer is robust, and otherwise the layer is called critical. In this section, we study layer robustness of commonly used neural networks on standard image classification benchmarks, in particular MNIST, CIFAR10 and ImageNet. All the networks are trained using SGD with Nesterov momentum, and piecewise constant learning rate schedule. Please refer to Appendix A for further details.
Finally, REF observed that some of the layers in trained neural networks are robust to re-initialization, but not to re-randomization.
Are All Layers Created Equal?
ArXiv
ArXiv
Computer Science Mathematics