Layer rotation: a surprisingly simple indicator of generalization in deep networks?
Abstract
Our work presents empirical evidence that layer rotation, i.e. the evolution across training of the cosine distance between each layer’s weight vector and its initialization, constitutes an impressively consistent indicator of generalization performance.
Compared to previously studied indicators of generalization, we show that layer rotation has the additional beneﬁt of being easily monitored and controlled, as well as having a networkindependent optimum: the training procedures during which all layers’ weights reach a cosine distance of 1 from their initialization consistently outperform other conﬁgurations -by up to 20% test accuracy.
Finally, our results also suggest that the study of layer rotation can provide a uniﬁed framework to explain the impact of weight decay and adaptive gradient methods on generalization.
Introduction
In order to understand the intriguing generalization properties of deep neural networks highlighted by [23, 34, 16], the identiﬁcation of numerical indicators of generalization performance that remain applicable across a diverse set of training settings is critical.
A well-known and extensively studied example of such indicator is the width of the minima the network has converged to [12, 16].
In this paper we present empirical evidence supporting the discovery of a novel indicator of generalization: the evolution across training of the cosine distance between each layer’s weight vector and its initialization (denoted by layer rotation ).
Indeed, we show across a diverse set of experiments (with varying datasets, networks and training procedures), that larger layer rotations (i e. larger cosine distance between ﬁnal and initial weights of each layer) consistently translate into better generalization performance.
In addition to providing an original perspective on generalization, our experiments suggest that layer rotation also beneﬁts from the following properties compared to alternative indicators of generalization:
• It is easily monitored and, since it only depends on the evolution of the network’s weights, can be controlled along the optimization through appropriate weight update adjustments
• It has a network-independent optimum (all layers reaching a cosine distance of 1) •
It provides a uniﬁed framework to explain the impact of weight decay and adaptive gradient methods on generalization.
In comparison, other indicators usually provide a metric to optimize ( e.g. the wider the minimum, the better) but no clear optimum to be reached (what is the optimal width?), nor a precise methodology to tune it (how to converge to a minimum with a speciﬁc width?).
By disclosing simple guidelines to tune layer rotations and an easy-to-use controlling tool, our work can also help practitioners get the best out of their network with minimal hyper-parameter tuning.
The presentation of our experimental study is structured according to three successive steps:
Development of tools to monitor and control layer rotation (Section 2); 2.
Systematic study of layer rotation conﬁgurations in a controlled setting (Section 3); 3.
Study of layer rotation conﬁgurations in standard training settings, with a special focus on SGD, weight decay and adaptive gradient methods (Section 4).
Related work is discussed in Supplementary Material.
Monitoring and controlling layer rotation
This section describes the tools for monitoring and controlling layer rotation during training, such as its relation with generalization can be studied in Sections 3 and 4.
2.1 Monitoring with layer rotation curves
Layer rotation is deﬁned as the evolution of the cosine distance between each layer’s weight vector and its initialization during training.
More precisely, let w tl be the ﬂattened weight tensor of the l th layer at optimization step t ( t 0 corresponding to initialization), then the rotation of layer l at training step t is deﬁned as the cosine distance between w t 0 l and w tl .
In order to visualize the evolution of layer rotation during training, we record how the cosine distance between each layer’s current weight vector and its initialization evolves across training steps.
We denote this visualization tool by layer rotation curves hereafter.
2.2 Controlling with Layca
The ability to control layer rotations during training would enable a systematic study of its relation with generalization.
Therefore, we present Layca (LAYer-level Controlled
Amount of weight rotation), an algorithm where the layerwise learning rates directly determine the amount of rotation performed by each layer’s weight vector during each optimization step (the layer rotation rates ), in a direction speciﬁed by an optimizer (SGD being the default choice).
Inspired by techniques for optimization on manifolds [1], and on spheres in particular, Layca applies layer-wise orthogonal projection and normalization operations on SGD’s updates, as detailed in Algorithm 1 in Supplementary Material.
These operations induce the following simple relation between the learning rate
ρ l ( t ) of layer l at training step t and the angle θ l ( t ) between w tl and w t − 1 l : ρ l ( t ) = tan ( θ l ( t )) .
Our controlling tool is based on a strong assumption: that controlling the amount of rotation performed during each individual training step (i.e. the layer rotation rate) enables control of the cumulative amount of rotation performed since the start of training (i.e. layer rotation).
This assumption is not trivial since the aggregated rotation is a priori very dependent on the structure of the loss landscape.
As will be attested by the inspection of the layer rotation curves, our assumption however appeared to be sufﬁciently valid, and the control of layer rotation was effective in our experiments.
A systematic study of layer rotation conﬁgurations with Layca
Section 2 provides tools to monitor and control layer rotation.
The purpose of this section is to use these tools to conduct a systematic experimental study of layer rotation conﬁgurations.
We adopt SGD as default optimizer, but use
Layca (cfr.
Algorithm 1) to vary the relative rotation rates (faster rotation for ﬁrst layers, last layers, or no prioritization) and the global rotation rate value (high or low rate, for all layers).
The experiments are conducted on ﬁve different tasks which vary in network architecture and dataset complexity, and are further described in Table 1.
3.1 Layer rotation rate conﬁgurations
Layca enables us to specify layer rotation rate conﬁgurations by setting the layer-wise learning rates.
To explore the large space of possible layer rotation rate conﬁgurations, our study restricts itself to two directions of variation.
First, we vary the initial global learning rate ρ (0) , which affects the layer rotation rate of all the layers.
During training, the global learning rate ρ ( t ) drops following a ﬁxed decay scheme (hence the dependence on t ), as is common in the literature (cfr.
Supp.
Mat.
A.6).
The second direction of variation tunes the relative rotation rates between different layers.
More precisely, we apply static, layer-wise learning rate multipliers that exponentially increase/decrease with layer depth (which is typical of exploding/vanishing gradients).
The multipliers are parametrized by the layer index l (in forward pass ordering) and a parameter α ∈ [ − 1 , 1] such that the learning rate of layer l becomes:
[Equation]
Values of α close to − 1 correspond to faster rotation of ﬁrst layers, 0 corresponds to uniform rotation rates, and values close to 1 to faster rotation of last layers.
Visualization of the layer-wise multipliers for different α values is provided in Supplementary Material.
3.2 Study of the relation between layer rotation and generalization
Figure 1a depicts the layer rotation curves (cfr.
Section 2.1) and the corresponding test accuracies obtained with different layer rotation rate conﬁgurations.
While each conﬁguration solves the classiﬁcation task on the training data ( ≈ 100% training accuracy in all conﬁgurations, cfr.
Supp.
Mat.), we observe huge differences in generalization ability (differences of up to 30% test accuracy).
More importantly, these differences in generalization ability seem to be tightly connected to differences in layer rotations.
In particular, we extract the following rule of thumb that is applicable across the ﬁve considered tasks: the larger the layer rotations, the better the generalization performance.
The best performance is consistently obtained when nearly all layers reach the largest possible distance from their initialization: a cosine distance of 1 (cfr. ﬁfth column of Figure 1a).
This observation would have limited value if many conﬁgurations (amongst which the best one) lead to cosine distances of 1 .
However, we notice that most conﬁgurations do not.
In particular, rotating the layers weights very slightly is sufﬁcient for the network to achieve 100% training accuracy (cfr.
third column of Figure 1a)).
Moreover, one could imagine training procedures with large layer rotations that do not generalize well, e.g. if large rotations are performed in a classiﬁcation on the training set ( 100% accuracy).
[Table]
random direction.
It is indeed necessary that the rotations performed coincide with improvements in the training error.
In particular, conﬁgurations with too high layer rotation rates can prevent training from happening, thereby escaping the scope of our rule of thumb (cfr.
Figure 3).
A study of layer rotation in standard training settings
Section 3 uses Layca to study the relation between layer rotations and generalization in a controlled setting.
This section investigates the layer rotation conﬁgurations that naturally emerge when using SGD, weight decay or adaptive gradient methods for training.
First of all, these experiments will provide supplementary evidence for the rule of thumb proposed in Section 3.
Second, we’ll see that studying training methods from the perspective of layer rotation can provide useful insights to explain their behaviour.
The experiments are performed on the ﬁve tasks of Table 1.
The learning rate parameter is tuned independently for each training setting through grid search over 10 logarithmically spaced values ( 3 − 7 , 3 − 6 , ..., 3 2 ), except for C10-CNN2 and
C100-WRN where learning rates are taken from their original implementations when using SGD + weight decay, and from [29] when using adaptive gradient methods for training.
The test accuracies obtained in standard settings are compared to the best results obtained with Layca, as provided in the 5th column of Figure 1a.
4.1 Analysis of SGD and weight decay
Figure 1b ( 1 st line) depicts the layer rotation curves and the corresponding test accuracies generated by SGD for each of the ﬁve tasks.
We observe that the curves are far from the ideal scenario disclosed in Section 3, where the majority of the layers’ weights reached a cosine distance of 1 from their initialization.
Moreover, in accordance with our rules of thumb, SGD reaches a considerably lower test performance than Layca.
Extensive tuning of the learning rate did not help SGD to solve its two systematic problems: 1) layer rotations are not uniform and 2) the layers’ weights stop rotating before reaching a cosine distance of 1.
Several papers have recently shown that, in batch normalized networks, the regularization effect of weight decay was caused by an increase of the effective learning rate [28, 13, 35].
More generally, reducing the norm of weights increases the amount of rotation induced by a given training step.
It is thus interesting to see how weight decay affects layer rotations, and if its impact on generalization is coherent with our rule of thumb.
Figure 1b ( 2 nd line) displays, for the 5 tasks, the layer rotation curves generated by SGD when combined with weight decay (in this case, equivalent to L 2 -regularization).
We observe that SGD’s problems are solved: all layers’ weights are rotated synchronously and reach a cosine distance of 1 from their initialization.
Moreover the observations conﬁrm our rule of thumb: the resulting test performances are on par with the ones obtained with Layca.
4.2 Analysis of adaptive gradient methods
The recent years have seen the rise of adaptive gradient methods in the context of machine learning ( e.g. RMSProp [27], Adagrad [6], Adam [17]).
Initially introduced for improving training speed, [29] observed that these methods also had a considerable impact on generalization.
Since these methods affect the rate at which individual parameters change, they might also inﬂuence layer rotations.
We will thus verify if their inﬂuence on generalization is coherent with our rule of thumb.
Figure 1c ( 1 st line) provides the layer rotation curves and test accuracies obtained when using adaptive gradient methods to train the 5 tasks described in Table 1.
We observe an overall worse generalization ability compared to Layca’s optimal conﬁguration and small and/or non-uniform layer rotations.
We also observe that the layer rotations of adaptive gradient methods are considerably different from the ones induced by SGD (cfr.
Figure 1b).
For example, adaptive gradient methods seem to induce larger rotations of the last layers’ weights, while SGD usually favors rotation of the ﬁrst layers’ weights.
Could these differences explain the impact of parameter-level adaptivity on generalization in deep learning?
In Figure 1c ( 2 nd line), we show that when
Layca is used on top of adaptive methods (to control layer rotation), adaptive methods can reach test accuracies on par with SGD + weight decay.
Our observations thus offer a novel perspective on adaptive gradient methods’ poor generalization properties, and provide supplementary evidence for our rule of thumb.
[Figure]
A CKNOWLEDGEMENTS
Thanks to the anonymous NeurIPS and ICLR reviewers for their helpful feedback on previous versions of this paper. Special thanks to the reddit r/MachineLearning community for enabling outsiders to stay up to date with the last discoveries and discussions of our fast moving ﬁeld. Simon and Christophe are Research Fellows of the Fonds de la Recherche Scientiﬁque – FNRS, which provided funding for this work.
A Supplementary Material
[Figure] 1 Code
Code to reproduce all the training procedures and ﬁgures of this paper is available at https://github.com/ anonymousDLresearch/Layer-rotation or at https://github.com/Simoncarbo .
A.2 Related work
After the intriguing generalization properties of deep neural networks were highlighted by [23, 34, 16], several works have tried to identify aspects of training which could predict generalization performance in a consistent and general way.
A ﬁrst line of work tries to identify the characteristics of trained models that correlate with good generalization properties.
Different complexity metrics have been proposed: norm-based metrics are studied in [2, 24, 19], sensitivitybased metrics in [31, 25] and sharpness-based metrics in [12, 16].
These three approaches are compared and further studied in [22].
Other works proposed complexity metrics that explicitly use the decomposition of deep neural networks in layers.
Such decomposition is also key to our work and has already been very successful when analysing the training difﬁculties of deep neural networks [3, 11, 8].
In [20], the similarity of hidden representations resulting from training with different initializations is used as an indicator of generalization.
In [21], sensitivity to perturbation of hidden representations is studied.
While the works described above help us understand the characteristics of models that generalize well, they don’t explicitly disclose how the training procedure itself leads to such characteristics.
A second line of work studies indicators of generalization that characterize the whole training trajectory instead of solely focusing on its endpoint.
The sharpness metric is revisited in [15] and [7] analyses stiffness.
While the presence of noise is believed to affect generalization, the mechanisms at play are poorly understood [9, 30] and a clear metric to quantify its inﬂuence on generalization is still lacking.
[14] studies the evolution of the euclidean distance between the model’s weight vector and its initialization in the context of large batch training.
This metric is the most similar to layer rotation, and also has the particularity of being both easy to monitor and to control.
Our work differs by the used distance metric (layer-level cosine distance instead of model-level euclidean distance) and by performing a more extensive study that extends the context of large-batch training.
A.3 Pseudocode for the Layca algorithm
Cfr.
Algorithm 1.
Algorithm 1 Layca, an algorithm that enables control over the amount of weight rotation per step for each layer through its learning rate parameter (cfr.
Section 2.2).
Require: o , an optimizer (SGD is the default choice)
Require: T , the number of training steps
L is the number of layers in the network for l=0 to L-1 do
Require: ρ l ( t ) , a layer’s learning rate schedule
Require: w l 0 , the initial multiplicative weights of layer l end for t ←while t < T do s 0 t , ..., s L − 1 t = getStep ( o, w 0 t , ..., w L − 1 t ) (get the updates of the selected optimizer) for l=0 to L-1 do s lt ← s lt − ( s lt
[Equation]
lt
[Equation]
lt
[Equation]
lt
[Equation]
w lt (project step on space orthogonal to w lt ) s
[Equation]
(rotation-based normalization)
[Equation]
(perform update) w
[Equation]
(project weights back on sphere) end for t ← t +end while
A.4 Visualizing the α parameter.
The α parameter is used in Section 3 to characterize the layer prioritization schemes used during training.
While the speciﬁc parametrization is provided in Equation 1, Figureprovides a graphical illustration of it.
A.5 Test accuracies for supplementary α and ρ (0) conﬁgurations
Cfr.
Figure 3.
A.6 Learning rate decay schemes
Our work uses standard learning rate decay schemes, as follows:
• C10-CNN1: 100 epochs and a reduction of the learning rate by a factor 5 at epochs 80, 90 and• C100-resnet: 100 epochs and a reduction of the learning rate by a factor 10 at epochs 70, 90 and• tiny-CNN: 80 epochs and a reduction of the learning rate by a factor 5 at epoch• C10-CNN2: 250 epochs and a reduction of the learning rate by a factor 5 at epochs 100, 170,• C100-WRN: 250 epochs and a reduction of the learning rate by a factor 5 at epochs 100, 170,
The only exceptions are C10-CNN2 and C100-WRN training with SGD+weight decay and with adaptive methods, where the learning rate decay schemes are the ones used in their original implementation or in [29].
A.7 Training errors associated to the layer rotation curves.
In Figures 1a, 1b and 1c, the test accuracies corresponding to each layer rotation curves visualization are provided.
While it is brieﬂy mentioned that training accuracy is close to perfect in most cases, Tables 2, 3 and 4 provide the exact values for completeness.
l/L 0 .0 .0 .0 .0 .1 .
M u l t i p li e r
Negative α 0 .0 .0 .0 .1 . l/L 0 .0 .0 .0 .0 .1 . M u l t i p li e r Positive α 0.
[Figure]
α = 0 .α = − 0 .ρ = 3 −ρ (0) = 3 −
Best
C
[Table] [Table]
[Table] [Table] [Table]