M AKING C OHERENCE O UT OF N OTHING A T A LL : M EASURING E VOLUTION OF G RADIENT A LIGNMENT
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose a new metric ( m -coherence ) to experimentally study the alignment of per-example gradients during training.
Intuitively, given a sample of size m , m -coherence is the number of examples in the sample that beneﬁt from a small step along the gradient of any one example on average.
We show that compared to other commonly used metrics, m -coherence is more interpretable, cheaper to compute ( O ( m ) instead of O ( m 2 ) ) and mathematically cleaner.
(We note that m -coherence is closely connected to gradient diversity, a quantity previously used in some theoretical bounds.)
Using m -coherence , we study the evolution of alignment of per-example gradients in ResNet and EfﬁcientNet models on ImageNet and several variants with label noise, particularly from the perspective of the recently proposed
Coherent Gradients (CG) theory that provides a simple, uniﬁed explanation for memorization and generalization [Chatterjee, ICLR 20].
Although we have several interesting takeaways, our most surprising result concerns memorization.
Naïvely, one might expect that when training with completely random labels, each example is ﬁtted independently, and so m -coherence should be close to 1.
However, this is not the case: m -coherence reaches moderately high values during training (though still much smaller than real labels), indicating that over-parameterized neural networks ﬁnd common patterns even in scenarios where generalization is not possible.
A detailed analysis of this phenomenon provides both a deeper conﬁrmation of CG, but at the same point puts into sharp relief what is missing from the theory in order to provide a complete explanation of generalization in neural networks.
I NTRODUCTION
Generalization in neural networks trained with stochastic gradient descent (SGD) is not wellunderstood.
For example, the generalization gap, i.e., the difference between training and test error depends critically on the dataset and we do not understand how.
This is most clearly seen when we ﬁx all aspects of training (e.g., architecture, optimizer, learning rate schedule, etc.) and vary only the dataset.
In a typical experiment designed to test this, training on a real data set (e.g., ImageNet) leads to a relatively small generalization gap, whereas training on randomized data (e.g., ImageNet with random labels) leads to a much larger gap (Zhang et al., 2017; Arpit et al., 2017).
The mystery is that in both cases (real labels and random)
the training accuracy is close to 100% which implies that the network and the learning algorithm have sufﬁcient effective capacity (Arpit et al., 2017) to memorize the training sets, i.e., to ﬁt an arbitrary mapping from the input images to labels.
But, what then, is the mechanism that from among all the maps consistent with the training set, allows SGD to ﬁnd one that generalizes well (when such a well-generalizing map exists)?
This question has motivated a lot of work (see e.g., Zhang et al.
; Arpit et al.
; Bartlett et al.
; Kawaguchi et al.
; Neyshabur et al.
Arora et al.
; Belkin et al.
Rahaman et al.
but no satisfactory answer has emerged.
As Nagarajan & Kolter (2019) point out, traditional approaches based on uniform convergence may not sufﬁce, and new ideas are needed.
A promising line of attack is via algorithmic stability Bousquet & Elisseeff (2002), but traditional stability analysis of SGD (e.g., Hardt et al.
; Kuzborskij & Lampert (2018)) does not account for the dataset, and without that, one cannot hope to get more than a vacuous bound.
Recently, a new approach called Coherent Gradients (CG) has been proposed that takes into account the training dataset in reasoning about stability (Chatterjee, 2020; Zielinski et al., 2020).
By analogy to Random Forests which also show dataset dependent generalization, CG posits that neural networks try to extract commonality from the dataset during the training process.
The key insight is that, since the overall gradient for a single step of SGD is the sum of the per-example gradients, it is strongest in directions that reduce the loss on multiple examples if such directions exist.
Intuitively, at one extreme, if all the per-example gradients are aligned we get perfect stability (since dropping an example does not affect the overall gradient) and thus perfect generalization.
At the other extreme, if all the per-example gradients are pairwise orthogonal, we get no stability (since dropping an example eliminates any descent along its gradient), and thus pure memorization.
This can be seen, for example, when trying to ﬁt a linear model y = w · x to the following dataset under the usual mean squared error loss:
[Equation]
Thus CG provides a simple, uniﬁed explanation for both memorization and generalization.
However, at the same time, CG leads to some basic empirical questions:
What does the alignment of per-example gradients,
i
e., coherence look like in practice?
As was noted in Chatterjee (2020), we expect a real dataset to have more coherence than a dataset with random labels, but how big is this difference quantitatively?
Is coherence in the random label case like that in the pairwise orthogonal case described above?
How does it vary with layer or architecture?
Is the coherence constant throughout training , or does it vary?
If so, how?
The key insight of CG (as described above) is a point-in-time observation, but in order to get a full picture of generalization we need to analyse the entire training trajectory.
For example, one might imagine that as more and more training examples are ﬁtted, coherence decreases, but is it possible for it to increase in the course of training?
In this paper, we propose a new metric called m -coherence to experimentally study gradient coherence.
The metric admits a very natural intuitive interpretation that allows us to gain insight into the questions above.
While we conﬁrm our intuitions in many cases, we also ﬁnd some surprises.
These observations help us formulate more precisely what is missing from the CG explanation for generalization, and thus point the way to future work in this direction.
P RIOR W ORK ON M ETRICS FOR E XPERIMENTALLY M EASURING C OHERENCE
Pairwise Dot Product.
An obvious starting point to quantify the alignment or coherence of a set of gradients is their average pairwise dot product.
Since this has a nice connection to the loss function, we start by reviewing the connection, and also set up notation in the process.
Formally, let D ( z ) denote the distribution 1 of examples from a ﬁnite 2 set Z , and assume without loss of generality that support( D )
=
Z .
For a network with d trainable parameters, let (cid:96) z ( w ) be the loss for an example z ∼ D for a parameter vector w ∈ R d .
For the learning problem, we are interested in minimizing the expected loss (cid:96) ( w ) :
= E z
∼D [ (cid:96) z
( w )] .
Let g z :=
[ ∇ (cid:96) z ]( w ) denote the gradient of the loss on example z , and g :=
[ ∇ (cid:96) ]( w ) denote the overall gradient.
From linearity, we have,
[Equation]
Now, suppose we take a small descent step h = − ηg (where η > 0 is the learning rate).
From the
Taylor expansion of (cid:96) around w , we have,
[Equation]
where the last equality can be checked with a direct computation.
Thus, the following are approximately equivalent:
• reduction in loss (due to a small step) divided by the learning rate, • squared (cid:96) 2 norm of the expected gradient, and, • expected pairwise dot product (where the expectation is over all pairs).
Example.
(Chatterjee, 2020) Consider a sample with m examples z i where 1 ≤ i ≤ m .
Let g i be the gradient of z i and further that (cid:107)
g
i (cid:107) =
(cid:107) u (cid:107) for some u .
If all the g i are the same, then g · g = (cid:107) u (cid:107) 2 .
However, if they are pairwise orthogonal, i.e., g i · g j = 0 for i (cid:54) =
j , then g · g = 1 m (cid:107) u (cid:107) 2 .
As this illustrates, the average expected dot product can vary signiﬁcantly depending on the coherence.
However, as a metric for coherence it is rather fragile.
For example, just re-scaling the loss can drastically alter the value of the metric.
Therefore, it can only be used to reason about coherence in very limited settings.
For e.g., Chatterjee (2020); Zielinski et al.
(2020) use it to verify that adding increasing amounts of label noise to a dataset reduces coherence but in order to do so they keep everything else the same, and limit their considerations to the start of training.
But, to study the evolution of coherence, even over a single training run requires normalization since the magnitude of the gradients changes signiﬁcantly in the course of training (e.g., see Appendix B).
Stiffness.
Fort et al.
in their preprint (2020) study two variants of the average pairwise dot product that they call sign stiffness and cosine stiffness .
In our notation these are
[Equation]
These are meant to capture how a small gradient step based on one input example affects the loss on a different input example.
Although Fort et al.
do not describe why they choose to transform the gradients in these speciﬁc ways, we expect it is to normalize the dot product so that it can be tracked in the course of training.
In their experience, they found sign stiffness to be more useful to analyze stiffness between classes whereas cosine stiffness was more useful within a class.
Gradient Confusion.
Sankararaman et al.
in their preprint (2019) introduce the notion of a gradient confusion bound.
The gradient confusion bound is ζ ≥
if for all z, z (cid:48)
∈
Z and z (cid:54)
=
z (cid:48) , we have, g z · g z (cid:48) ≥ − ζ .
They use this concept to study theoretically the convergence rate of gradient descent, but in their experimental results they measure the minimum cosine similarity between gradients, i.e.,
[Equation]
We note that the non-linearities (and to a lesser extent the z (cid:54) =
z (cid:48) restriction) make it hard to tie stiffness or minimum cosine similarity to what happens during training; speciﬁcally, to the change in the loss function as a result of a gradient step which is the expectation over all per-example gradients.
A N EW M ETRIC FOR C OHERENCE
The key insight behind our proposal is that there is a natural scaling factor that can be used to normalize the expected dot product of per-example gradients (i.e., the quantity in (1)) that preserves the connection to the loss.
Consider the Taylor expansion of each individual loss (cid:96) z around w when we take a small step h z down its gradient g z :
[Equation]
Taking expectations over z we get,
[Equation]
The quantity in (2) has a simple interpretation: It is the reduction in the overall loss (cid:96) if each example (cid:96) z could be optimized independently.
As might be expected intuitively, it is a lower bound on the quantity in (1) and is tight when all the per-example gradients are identical.
We prove this formally in §4.
Thus, it serves as a natural scaling factor for the expected dot product, and we obtain a normalized metric for coherence (denoted by α ) from (1) and (2):
[Equation]
Thus, α is the change in the overall loss due to a small gradient step as a fraction of the maximum possible change in loss if each component of the loss could be optimized independently .
As noted before, 0 ≤ α ≤ 1 , and the maximum is achieved when all the gradients are identical, and the minimum is achieved when the expected gradient is 0, i.e., a stationary point is reached.
A natural scale for α .
Once again, consider a sample with m examples
z i where 1 ≤ i ≤ m .
Let g i be the gradient of z i .
Suppose further that the g i are pairwise orthogonal i.e. g i · g j = 0 for i (cid:54) =
j .
It is easy to check that α = 1 /m .
For a sample of size m , we call this value of α the orthogonal limit .
Since in the orthogonal case, each example is optimized independently, going down the expected gradient is 1 /m times as slow as optimizing each independently.
If the gradients are better aligned, we expect
Example (Commonality).
For 1 ≤ i ≤ m , suppose each g i has a common component c and an idiosyncratic component u i , i.e., g i
=
c + u i with u i · u j = 0 for 1 ≤ j ≤ m and j (cid:54)
=
i ; u i · c = 0 ; and say, u i · u i = (cid:107) u (cid:107) 2 for some u .
It is easy to see that α in this case is 1 m [1 + ( m − 1) · f ] where f = (cid:107) c (cid:107) 2 / ( (cid:107) c
(cid:107)
+ (cid:107) u (cid:107) 2 ) .
These examples along with the observation that 0 ≤ α ≤ 1 suggests a more evocative (even if less accurate and less general) interpretation: In a given sample, α is the average fraction of examples that each example helps or supports.
Thus, when analyzing experimental data, for a sample of size m , it is convenient to deﬁne a new quantity m -coherence as follows:
[Equation]
Thus m -coherence in the orthogonal limit is 1 and in the identical case is m .
Intuitively, m -coherence of a sample is the number of examples (including itself) that any one example helps on average.
Advantages.
α and m -coherence have several advantages over the metrics discussed in §2:
• Computational Efﬁciency.
For a sample of size m , due to (3), α can be computed exactly in O ( m ) time in contrast to O ( m 2 ) time required for stiffness and cosine dot products.
Furthermore, it can be computed in a streaming fashion by keeping two running sums, so the per-example gradients need not be stored.
Thus, in our experiments we are able to use sample sizes a couple of orders of magnitude higher than those in Fort et al.
(2020) and Sankararaman et al.
• Mathematical Simplicity.
We believe our deﬁnition is cleaner mathematically.
This
allows us to reason about the metric more easily.
For example
We can show that the coherence of minibatch gradients is greater than that of individual examples (Corollary 3.1).
Therefore, care must be taken if minibatch gradients are used in lieu of example gradients in computing coherence (e.g., as in Sankararaman et al.
Explicitly ruling out z (cid:54) =
z (cid:48) as in done in stiffness and cosine similarity to eliminate self-correlation is unnatural and can get tricky in practice due to near-duplicates or multiple examples leading to same or very similar gradients.
We obtain meaningful values without imposing those conditions, but if one insists on removing self-correlations, then subtracting 1 /m from α or 1 from m -coherence is a more principled way to do it.
The non-linearities in stiffness and cosine similarity amplify small per-example gradients potentially overstating their importance, and lead to a discontinuity (or undeﬁned behavior) with zero gradients.
However, we can cleanly account for the effect of negligible gradients in our observations (e.g., see Lemma 4).
• Interpretability.
Finally, as discussed in detail above, they are normalized and yet easily interpretable due to the natural connection with loss.
Prior Work on Gradient Diversity.
While writing this paper we discovered that the reciprocal of α appears in the theory literature as gradient diversity .
This was used by Yin et al.
(2018) in theoretical bounds to understand the effect of mini-batching on convergence of SGD.
(A similar result appears for least squares regression in Jain et al.
They show that the greater is the gradient diversity, the more effective are large mini-batches in speeding up SGD.
Although they support their theoretical analysis with experiments on CIFAR -10 (where they replicate 1 /r of the dataset r times and show that greater the value of r less the effectiveness of mini-batching to speed up) they never actually measure the gradient diversity in their experiments (or further study its properties).
Also, note that for our purposes α is a better choice than 1 /α – not just because coherence rather than incoherence is what leads to generalization – but also since the latter can diverge: g can be 0 without all g z being zero (e.g., at the end of training in an under-parameterized setting).
A M ORE G ENERAL S ETTING FOR C OHERENCE AND S OME B ASIC F ACTS
Our notion of coherence is not speciﬁc to gradients (or optimization) but extends naturally to vectors in Euclidean spaces.
Let V be a probability distribution on a collection of m vectors in an Euclidean space.
In accordance with (3), we deﬁne the coherence of V (denoted by α ( V ) ) to be
[Equation]
Note that E [ v · v ] = 0 implies E [ v · v (cid:48) ]
= 0 .
In what follows, we ignore the technicality of the denominator being 0 by always assuming that there is at least one non-zero vector in the support of V (which also held in our experiments).
We list some basic facts and the proofs are in Appendix A.
Theorem 1 (Boundedness) .
We have 0 ≤ α ( V ) ≤ 1 .
In particular, α ( V ) =
iff E v ∼V [ v ] =
and α ( V )
=
iff all the vectors are equal.
Lemma 2 (Scale Invariance) .
For non-zero k ∈ R , let k V denote the distribution of the random variable kv where v is drawn from V .
We have α ( k V ) =
α ( V ) .
Theorem 3 (Stylized mini-batching) .
Let v 1 , v 2 , .., v k be k i.i.d.
variables drawn from V .
Let W denote the distribution of the random variable w = 1 k (cid:80) ki
=1 v i .
We have,
[Equation]
Corollary 3.1 (Minibatch ampliﬁcation) .
α ( W ) ≥ α ( V ) with equality
iff α ( V ) =
or α ( V ) = 1 .
Remark.
This formulation provides a nice perspective on the type of results proved in Yin et al.
(2018) and Jain et al.
When α (cid:28) 1 /k but non-zero (i.e., we have high gradient diversity), creating mini-batches of size k increases coherence almost k times.
But, when α ≈ 1 (i.e., low diversity) there is not much point in creating mini-batches since there is little room for improvement.
Lemma 4 (Effect of zero gradients) .
If W denotes the distribution where with probability p > 0 we pick a vector from V and with probability 1 − p we pick the zero vector then α ( W )
=
p · α ( V ) .
Example (Coherence Reduction).
If we add k zero gradients to the collection of gradients con- structed in the example of §3 (Commonality), using Lemma 4, we get,
[Equation]
where n = m + k is the size of this new sample.
For a ﬁxed n , as k increases, α decreases going down to 1 /n
(the orthogonal limit) when all but one vector in the sample is zero, i.e., k = n − 1 .
E XPERIMENTAL R ESULTS
As stated in the introduction, our objective in developing m -coherence is to better understand the recently proposed CG explanation for generalization (Chatterjee, 2020; Zielinski et al., 2020). We [Figure] m - c
o
h e r e n c e ( c l a ss i f
i c a t i o n l a y e r )
[Figure]
e r e n c
[Figure]
[Figure]
i dd l e l a y e
[Figure]
o
h e r e n c e
( f i r s t l a y e r )
[Figure]
alignment of per-example gradients of a ResNet-18 during training as measured with m -coherence on samples of size m = 50 , 000 on 3 variants of ImageNet with different amounts of label noise.
Our main ﬁnding is that coherence not only decreases in the course of training (as expected from Lemma 4 when examples get ﬁt), but it also increases.
A high peak is reached rapidly with real labels the and a much lower is reached slowly many now measure m -coherence in a setting very similar to that used by Zielinski et al.
(2020) for studying memorization and generalization.
Methodology.
We train ResNet-18 (He et al., 2016) models on ImageNet with original labels (0% noise), and two derived datasets: one with half the training labels randomized (50% noise), and another with all the training labels randomized (100% noise).
3 We use SGD with momentum (0.9), a batch size of 4096, and the learning rate schedule proposed in Goyal et al.
We turn off augmentation and weight decay to observe memorization in the noisy cases within a reasonable number of steps.
For each dataset, we track m -coherence on a random (but ﬁxed) set of m = 50 ,training examples.
Figure 1 shows the data from our experiments.
Each column corresponds to a different experiment and the rows show loss, accuracy, and m -coherence for the entire model and some speciﬁc layers.
Real Labels.
Our ﬁrst experiment (shown in the second column of Figure 1 for reasons that will become clear shortly) measures the m -coherence (row 3) for training with 0% noise, i.e., the real
ImageNet labels at the end of each epoch.
The coherence at the end of the ﬁrst few epochs is very high, between 10 3 and 10 [-
-]4 and it decreases as more training examples get ﬁt.
We note that although there is some ﬂuctuation in the coherence, it stays high (above 10 2 and often above 10 3 ) until well after the accuracy crosses the 50% mark.
It settles at 1 after all the examples are ﬁt.
The high coherence in early epochs agrees well with the intuition from CG that real datasets have good per-example gradient alignment since that is what is necessary for good generalization as per the theory.
The subsequent decrease in coherence in the course of training is expected from Lemmaunder the assumption that the gradients of ﬁtted examples become small.
Random Labels.
Our second experiment (column 6) shows that with random labels, the coherence in the ﬁrst few epochs is low (between 1 and 10).
It increases steadily until it reaches a peak in epochs 40 to 60 (above 10 2 but less than 10 3 ) followed by a decrease.
The low coherence (near the orthogonal limit) in the ﬁrst few epochs agrees well with CG as discussed in the introduction but the subsequent increase is surprising (though not in contradiction with CG as discussed later).
The increase is not small since at its peak each example is helping hundreds of other examples (though it is well below the 10 4 peak seen with real labels).
Once again, as examples get ﬁtted, coherence decreases as expected from Lemma 4, though not back down to 1, likely since our training only goes on till about 80% accuracy is reached.
The increase in one case and not the other leads to a natural question with implications about the dynamics of SGD: Is the evolution of coherence fundamentally different between the well-generalizing case (real labels) and the memorization case (random labels)?
Coherence in the First Epoch.
To study this question, we took a closer look at the ﬁrst epoch.
We recorded m -coherence at initialization (i.e., before the ﬁrst step) and, thereafter, for every other step in the epoch.
Since this requires computing the per-example gradients for 50K examples every 2 steps, this was our most computationally expensive experiment taking a day per run using TPUs.
The results are shown in columns 1 (real labels) and 5 (random).
[Figure]
ﬁnd that in both cases coherence starts from about and rising.
is much faster for real labels than for random labels.
We ran additional experiments with 50% noise (column 3) and with 25% and 75% noise to conﬁrm that the slope with which coherence increases depends inversely on the label noise
The Overall Evolution.
If we combine the data from the steps of the ﬁrst epoch with the rest of training (e.g., in row 3 we jointly view columns 1 and 2), we ﬁnd a remarkably consistent pattern across all noise levels: m -coherence follows a broad parabolic trajectory (albeit with some local variation and noise) where it starts small, increases to a peak, and then goes back down.
Thus, there is always an initial increase in coherence, just on different timescales.
From this point of view, the evolution of coherence in the memorization case does not look fundamentally different from that in the well-generalizing case.
Impact of Layers.
The bottom 3 rows of Figure 1 shows coherence by layer for 3 illustrative layers (the ﬁrst convolution layer, a convolution layer in the middle, and the ﬁnal fully connected layer).
Although the speciﬁc values are different across the layers, 4 we notice that the broad trajectory observed for the coherence of the entire model holds for each individual layer.
Thus, the trajectory (and in particular the increase) is not driven by one speciﬁc layer.
Impact of Architecture.
We studied one other architecture, EfﬁcientNet (Tan & Le, 2019), and found a similar increase in coherence and indeed a similar broad trajectory (Figure 2 and Appendix B).
Coherence on Test Set.
For completeness, we also measured the coherence on m examples not used for training (drawn from the ImageNet validation set).
We defer the discussion to the Appendix.
Reconciliation with Other Studies.
Although a direct comparison with the results in Fort et al.
(2020) and Sankararaman et al.
(2019) is not possible due to differences in metrics, sample sizes, datasets (ImageNet v/s CIFAR ), and effects studied; in as much as they can be compared, we did not ﬁnd contradictions.
Please see Appendix B for more details.
D ISCUSSION AND F UTURE W ORK
In this paper, we have presented m -coherence , a mathematically clean and easily interpretable metric to study the evolution of the alignment of per-example gradients in the course of training.
As stated in the introduction, our motivation in developing this metric is to better understand the recently proposed
CG explanation for generalization (Chatterjee, 2020; Zielinski et al., 2020). As noted in Sections 2 and 3, in contrast to unnormalized metrics such as the expected pairwise dot products of per-example gradients (which are difﬁcult to compare across different training steps or architectures), or previously proposed normalized metrics such as stiffness or confusion (which are difﬁcult to interpret due to ad hoc normalization that breaks the connection with the loss)
, the natural normalization employed in m -coherence allows us to directly interpret the value of m -coherence as the number of examples (including itself) that any one example helps on average. In turn, this property permits us to reason directly about the CG explanation.
Insights on Coherent Gradients (CG).
Based on our measurement of coherence in the setting of
Zielinski et al. (2020), we can now pinpoint precisely what is missing in the original CG explanation for generalization: CG assumes that coherence exists, and from that, it shows how gradient descent can exploit it—this is the causal mechanism that the CG papers verify through intervention experiments.
However, CG does not address where the coherence comes from and simply assumes that it is a property of the dataset.
For example, CG assumes that real data has high coherence, and that data with random labels has low coherence (e.g., Section 2.2 in Chatterjee (2020)).
Our measurements show that this assumption about coherence is not entirely true:
For both real and random labels, at the start of training, coherence is low, and then increases.
That said, consistent with the intuition from CG, we ﬁnd that the peak coherence with real labels is much higher (each example helps ≈ 10 4 other examples) than the peak with random labels (each example only helps ≈ 10 2 other examples).
Furthermore, the peak with real labels is reached very quickly (well within the ﬁrst epoch)
whereas with random labels it takes much longer (many epochs). The failure of this assumption leads to a very interesting open question: What causes the alignment of per-example gradients, i.e., coherence to increase in the course of training?
Since the increase happens both in the case of real labels and in the case of random labels, we conjecture that the increase is an optimization phenomenon and not directly related to generalization.
Going back to the analogy with random forests in the introduction, the creation of coherence is similar to the ﬁnding of commonality (possibly spurious) between examples during decision tree construction (c.f. discussion on generalization in deep learning in Section 4 of Chatterjee & Mishchenko (2020)).
In other words, while an explanation for why gradient descent generalizes is provided by the original CG hypothesis (if coherence exists, gradient descent exploits it by taking a larger step in the direction that improves many examples), we now need an explanation for the optimization phenomenon of why coherence increases in gradient descent (since it does not start off high even in the case of real data). It is important to note that in contrast with the increase in coherence which is a mystery, the decrease in coherence as training progresses, and more examples get ﬁt, is expected, as may be observed from
Lemma 4 and the example on Coherence Reduction. Practical Applications. Finally, it would be interesting to explore some practical applications of this metric. For example, we could consider using a metric derived from m -coherence to predict the generalization gap as a result of training.
While a simple heuristic might be to use the area under the curve of m -coherence over time as an (inverse) measure of the generalization gap, a better heuristic would also incorporate for the distance travelled in parameter space since, as our experiments show, low coherence for relatively few steps (e.g., early in training with real labels) or when gradients are small (e.g., late in training) is not fatal for generalization. Another possible use of m -coherence could be to freeze certain layers if they have low coherence to prevent overﬁtting in those layers or to improve the speed of training (e.g. if all lower layers are frozen). These practical applications would require even faster methods to estimate coherence. One way to do that might be to compute the coherence between mini-batches (with exponential moving averages) and then applying Corollary 3.1 in reverse.
A O MITTED P ROOFS
We present the proofs omitted from Section 4.
Recall from (4) that
[Equation]
Theorem 1 (Boundedness) .
We have 0 ≤ α ( V ) ≤ 1 .
In particular, α ( V ) =
iff E v ∼V [ v ] =
and α ( V )
=
iff all the vectors are equal.
Proof.
Since v · v ≥ 0 for any v , we have E v ∼V [ v · v ] ≥ 0 .
Furthermore, it is easy to verify by expanding the expectations (in terms of the vectors and their corresponding probabilities) that
[Equation]
Therefore, α ( V ) ≥ 0 .
Likewise, another direct computation shows that
[Equation]
Since from Equation 6 we have E [ v ] · E [ v ] =
E [ v · v (cid:48) ] , it follows that α ( V ) ≤ 1 .
Furthermore, since each term of the expectation on the left is non-negative, equality is attained only when all the vectors are equal.
Lemma 2 (Scale Invariance) .
For non-zero k ∈ R , let k V denote the distribution of the random variable kv where v is drawn from V .
We have α ( k V ) =
α ( V ) .
Proof.
[Equation]
Theorem 3 (Stylized mini-batching) .
Let v 1 , v 2 , .., v k be k i.i.d.
variables drawn from V .
Let W denote the distribution of the random variable w = 1 k (cid:80) ki
=1 v i .
We have,
[Equation]
Proof.
The ﬁrst equality follows from Lemma 2.
For the second equality, we have,
[Equation]
By dividing the numerator and denominator of the last expression by k E v ∼V [ v · v ] the required result follows.
Corollary 3.1 (Minibatch ampliﬁcation) .
α ( W ) ≥ α ( V ) with equality
iff α ( V ) =
or α ( V ) = 1 .
Proof.
From the previous theorem, the transformation in coherence due to stylized mini-batching is given by the map α (cid:55)→
[Equation]
Now, since α ≤ 1 , we have k ≥ 1 + ( k − 1) · α , and since α ≥ 0 , multiplying both sides by
[Equation]
· α we have
[Equation]
Finally, it is easy to check that the only two ﬁxed points of the map are α = 0 and α = 1 .
Lemma 4 (Effect of zero gradients) .
If W denotes the distribution where with probability p > 0 we pick a vector from V and with probability 1 − p we pick the zero vector then α ( W )
=
p · α ( V ) .
Proof.
[Equation]
B A DDITIONAL E XPERIMENTAL R ESULTS AND D ISCUSSION
Experimental Setup.
Our code for running experiments was heavily based on an open source
Tensorﬂow example, 5 with modiﬁcations to allow label randomization and coherence metric logging.
We used SGD with momentum (0.9), a batch size of 4096, and the learning rate schedule proposed in Goyal et al.
We did not use weight decay or random augmentation of the input.
When measuring per-example gradients to compute m -coherence, for normalizing activations in a batch norm (BN) layer, we use the moving averages collected by the layer, i.e., we measure perexample gradients in eval mode.
However, in the ﬁrst few steps of training, since the moving averages are initialized at 0, the per-example gradients measured in this manner may not be accurate.
We address this by priming the moving averages in the BN layers by running 40 steps without updating any trainable parameters (or momentum values) with a no-op optimizer at the start of our experiments.
The importance of this correction depends upon the architecture.
For example, for ResNeton ImageNet, without this correction, we see an artiﬁcially high loss (16 instead of the expected ln(1000)
≈ 6 . 9 )
in the ﬁrst few steps of training due to very large activations feeding the ﬁnal softmax layer.
This leads to artiﬁcially high gradient measurements and m -coherence.
However, these measurement artifacts disappear in about 25 steps as the BN averages get primed.
Variation across Layers.
One advantage of m -coherence is that it is natural to use it to compare different projections of the per-example gradients and as such can be used to directly compare different layers with each other.
Rows 4, 5 and 6 of Figure 1 shows the m -coherence of the classiﬁcation layer, a convolution layer in the middle and the ﬁrst convolution layer respectively.
We only show the m -coherence for the weights (since in a spot check the m -coherence for weights and biases for a given layer looked very similar).
We make a few observations.
First, convolutional layers have higher coherence than the fully connected layer.
Although this could be a function of depth, we note that this may also be expected for a different reason.
Since convolutional layers have ﬁlters that are instantiated at multiple sites, and the gradients from those sites for a single example add up in the overall gradient for a single gradient.
Therefore, by reasoning similar to that of Lemma 3.1, we expect the coherence of the gradient across sites to be greater than those for the sites individually.
And with more sites, we expect greater coherence.
This is another way to see that weight sharing prevents overﬁtting.
Second, the convolutional layers, particularly the ﬁrst one shows high coherence for random labels (though still generally lower than those for real labels).
However, in the fully connected layer, there is a much greater difference between real and random labels.
For random, it is generally quite low reaching 10 at the peak, whereas for real, while the examples are being ﬁt, it is usually above 10 and in early epochs, before overﬁtting sets in, it even exceeds 100.
Third, the only place where m -coherence sometimes falls below 1 (the orthogonal limit) is in the ﬁrst convolutional layer for real data after training accuracy has reached 100%.
At that point, the layer may be over-constrained, i.e., improving the loss on one example may degrade it on another (though see the discussion below on test set coherence).
Everywhere else, m -coherence tends to be at or above the orthogonal limit in line with our expectation that this learning problem is over-parameterized.
Finally, we note that the different layers for EfﬁcientNet in Figure 3 show similar characteristics as those for ResNet.
[Figure]
m - c
o
h e r e n c e ( c l a ss i f
i c a t i o n l a y e r )
[Figure]
m - c
o
h e r e n c e
m i dd l e l a y e r )
[Figure]
m - c
o
h e r e n c e
[Figure]
f i r s t l a y e r )
[Figure]
Coherence on Test Set.
For completeness, we also measured the coherence on m examples not used for training (drawn from the ImageNet validation set).
They are shown in columns 1 and 2 of Figureas “test.”
In the ﬁrst epoch, we ﬁnd that test and training coherence are roughly similar.
However, when we look at the rest of training, we ﬁnd that in the early part of the rest, test coherence is below that of training coherence, but in the later part, the opposite is true.
This may be further evidence that coherence creation is a pure optimization phenomenon (as per the discussion in Section 6 of the main paper), i.e., the coherence creation (and subsequent consumption) is speciﬁc to the training examples.
It is interesting to observe that particularly for the convolutional layers, at the end of training, the test m -coherence is at 10 whereas training m -coherence is at 1 or even lower.
This suggests that those layers are adapted to the training examples beyond what is likely to generalize.
Reconciliation with Other Studies.
Fort et al.
(2020) use the cosine and sign stiffness measures to study how gradient alignment depends on class membership, distance in input space between data [Figure] m - c o
h e r e n c e ( c l a ss i f i c a t i o n l a y e r ) [Figure] m - c
o
h e r e n c e
m i dd l e l a y e r )
TrainingTest
[Figure]
h e r e n c e
[Figure]
a y e r )
[Figure]
points, training iteration and learning rate.
They use MNIST , Fashion MNIST , CIFAR -10/100 and
MNLI datasets.
Typical sample sizes are around 500 (for the 10 label datasets) and 3000 (for thelabel datasets).
They do not study label noise or memorization explicitly.
In their class-based analysis, they ﬁnd that initially, an example of a class only helps other examples in its class and adversely impacts examples of other classes.
However, in the course of training, this effect goes down, and stiffness between classes goes up (though only to end up at 0).
We do not explicitly perform a class-based analysis, since with 1000 classes and about 1.2M training examples in ImageNet, we expect on average only 2 to 3 examples in each class pair.
However, implicitly, our study is an inter-class analysis (though not a class-pair analysis) since in our sample, each example is expected to see roughly 1000 times as many examples of other classes as it does its own class.
Our results indicate that examples in one class do help examples in other classes at different points in training since m -coherence is often in 1000s, and in a sample of approximately 50 k , we expect only about 50 examples per class.
However, since our metric is very different (as discussed in detail in Sections 2 and 3 of the paper), and the error bars in their study are large (as indicated in their Figure 5), we do not directly see any contradictions in the experimental data between their study and ours.
Finally, we do not study coherence as a function of input distance between examples or of learning rate changes, though we are interested in investigating the latter in future work.
Sankararaman et al.
(2019) show theoretically that high gradient confusion impedes the convergence of SGD, and also analyze how factors such as network depth and width and initialization impact gradient confusion.
They validate their theoretical results with experiments on MNIST , CIFAR -10, and CIFAR -
(real labels only, since they do not study memorization) where they measure the minimum cosine similarity (MCS) between different training examples.
6 They mainly focus on the
MCS value at the end of training as various architectural parameters are varied, but in Figures 7(c) and 8(c) in the appendix, they show the trajectory during training.
There, we ﬁnd that MCS starts low, increases to a peak and then comes back down again, in qualitative agreement with our ﬁndings.
Finally, in Figure 5, we show the individual terms of α (the numerator E [ g z · g z (cid:48) ] and denominator E [ g z · g z ] ).
Although the numerator can be estimated from the slope of the loss curve (as per equation (1) in the main paper), without the denominator to give it scale, it is hard to understand what variations are meaningful.
As an extreme example, we see that, as expected from the loss curve (and this may be seen in the loss plots from other studies such as Zhang et al.
; Zielinski et al.
, in the 100% random case, for the ﬁrst 20 epochs or so, the norm of the expected gradient is close to zero (Figure 5, last column, row 1).
However, there is signiﬁcant activity in the denominator (row 2; and this is not typically recorded in experiments).
By considering the quotient, and furthermore, by putting it into context with the orthogonal limit (as we do with m -coherence where that limit sets the scale), we can see that there is a deﬁnite build up in coherence in that period (row 3).
Effect of sample size m .
Figure 6 shows the effect on m -coherence and α of doubling m from our baseline value of 50 , 000 to 100 , 000 .
We see that the numerical values are generally the same, showing a slightly upward bias in m -coherence for the larger value of m as might be expected.
25% and 75% label noise.
The data for ResNet-18 training on 25% and 75% label noise is shown in Figure 7.
This conﬁrms the pattern noted in the main paper (Section 5) that with increasing noise, there is a decrease in the rate at which coherence increases.
[Figure]
∼ D , z
∼
D [ g z · g z 0 ] ( c l a ss i f i c a t i o n l a y e r )
[Figure]
g z ] ( c l a ss i f i c a t i o n
[Figure]
[Figure] [Figure]
[Figure]
l a ss f i a t i o n l a y r )
[Figure]
[Figure] [Figure]
D [ g z · g z 0 ] ( f i r s t l a y e r )
[Figure]
∼
[ g z · g z ] ( f i r s t l a y e r )
[Figure]
( f i r s t l a y e r )
[Figure]
corresponding to Figure 1.
Note that even when the expected gradient is ﬂat (as may be inferred even from the slope of the loss function), there is activity in the denominator which gets picked up with α or m -coherence particularly, if the scale is set appropriately w.r.t.
to the orthogonal limit.
[Figure] [Figure]
m - c
o
h e r e n c e ( e n t
i r e m o d e l )
[Figure]
- c o
[Figure]
e r e n c e ( c l a ss i f
i c a t i o n l a y e r )
[Figure]
m - c
[Figure]
r e n c e ( m i dd l e l a y e r )
[Figure]
c o h e r e n c e
( f i r s t l a y e r )
[Figure]