Paper: Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimization (ICLR-2024)

Reviewer comment (Writing):
The writing can be organized in a better way. There are many claims that are buried in long paragraphs. It is suggested that the authors number and list their claims/hypotheses/conjectures clearly, and then provide explanations for each of the claims. Even if there is no conclusive evidence, a clear statement should be presented so that future work can cite the statement unambiguously.

Author response:
> I would suggest that the authors number and list their claims/hypothesis/conjectures clearly, and then provide explanations for each of the claims.

Thanks for this suggestion: **we agree that this would improve clarity, and we will go through to highlight individual statements.** We imagine you may have been referring primarily to Sections 3.1 and 3.2, but please let us know if you had in mind any specific claims that you felt were not sufficiently clear.

Reviewer comment (Evaluation):
Currently there are no clear quantitative measurements. It is suggested that the authors provide one or two measurements to quantify "opposing signals" so that later work can examine their claims more precisely.

Author response:
> I do suggest that the authors provide one or two measurements to quantify "opposing signals"

We did our best to provide quantitative measurements, such as the gradient norm/curvature for individual samples (Figure 2) and the class logits/probabilities and embedding norms of “pure” color features (Figure 4 + appendix). These metrics are with respect to training points, since we are unsure how one could quantitatively measure the more conceptual idea of a “feature” / “signal”. Additionally, reviewer u2Ni suggested that we plot the distribution of changes in loss for each point over several iterations, which we have added as Figure 31 in Appendix D.

Reviewer comment (Evaluation):
My main concern with the current state of the submission is that the notion of outlier training samples does not seem to be precise enough and might require further clarification. For example, the authors show that the loss of almost half of the training samples may go up during a single iteration, but this change is more significant for a smaller number of outlier samples. It is not clear what fraction of the training samples are outliers and more generally what the distribution of the loss change over all samples is.

Author response:
Thank you for your thoughtful comments! We are very much in agreement that our finding “can have significant impacts on our understanding of the optimization dynamics of deep neural networks”. We hope our discussion below sufficiently addresses your remaining questions.

> “the notion of outlier training samples does not seem to be precise enough and might require further clarification.”

We agree that having a more precise definition of an “outlier”, beyond the definition we use in our theoretical analysis in Section 3.3, would be ideal. Unfortunately, such a concept is inherently fuzzy, as our primary intent was to convey something qualitative with respect to outliers in *image* space. **In our next point we incorporate your suggestion re: outliers in loss space**, but first we want to clarify that we intentionally do not try to define an outlier in image space, since this would require committing to an incomplete definition. To make this point more explicit, we have added a subsection to Section 2 (highlighted in blue). Please let us know if you have any suggestions for how this discussion could be improved.

> “It is not clear…what the distribution of the loss change over all samples is.”

For outliers with respect to *change in loss*, we think your idea to consider the entire distribution is a good one! We did visualize this early in our experiments: it is consistently concentrated in a nice bell curve centered near 0. **Per your suggestion, we’ve produced a plot of this distribution for a few iterations and added it to Appendix D (Figure 31).** The mean is indeed very close to 0 (typically  around $-.01$) and the standard deviation is usually around 0.3 to 0.5, sometimes a bit more, but never more than 0.8. The fraction of samples which are more than 2 (or 3 or 4, respectively) sds above/below the mean is consistently around .055 (.015 or .006, respectively). Thus the points which are changing in loss by ~1.0 or more (those depicted in Figure 1) are ~5% or less of the data, which we believe is reasonable to consider an outlier. **Thanks for this idea, we’ll add further discussion of this plot in the main body.**

Reviewer comment (Theory):
The relationship between the theoretical statements and the empirical observations of the paper is not entirely clear to me. The theory seems concerned with analyzing the sharpness throughout training with *gradient flow* which monotonically decreases the training loss, while a core part of the observation is to justify the spikes in the training loss that occur due to *large step size gradient descent*.

Author response:
Thank you for your thoughtful comments! We are very much in agreement that our finding “can have significant impacts on our understanding of the optimization dynamics of deep neural networks”. We hope our discussion below sufficiently addresses your remaining questions.

> “The theory seems concerned with analyzing the sharpness throughout training with gradient flow which monotonically decreases the training loss”

We think our introduction of the analysis was not as clear as it could have been, and we appreciate you pointing out this confusion. Our analysis tracks the trajectory of gradient flow from the start of training up until reaching the edge of stability. Until reaching this point, the paths of GF and GD on neural networks are quite similar, particularly for small timescales (see [1])---therefore, this analysis tracks the GD path closely for “reasonable” learning rates $\eta$. The purpose of our analysis up to this point is to show that after the initial decrease, the sharpness will monotonically increase up to *at least* $\frac{2}{\eta}$, usually much more. On real data, it is empirically well-established that gradient descent will start to oscillate when this threshold is crossed. We do not prove that this also happens on our model, but we verify it with experiments in Appendix D. Our result also shows specifically that this oscillation will occur in $b_o$, the part of the network which determines how the opposing signal is used, and this bears out in our experiments.

**We have made this point more explicit in a new paragraph at the beginning of Section 3.3.** Please let us know if you have suggestions for how to make this more clear!

Reviewer comment (Evaluation):
As mentioned above, it might help the readers have a better understanding if there is a visualization of the distribution of the change in loss for training samples, highlighting the frequency of the outliers in comparison with the entire training set.

Author response:
Thank you for your thoughtful comments! We are very much in agreement that our finding “can have significant impacts on our understanding of the optimization dynamics of deep neural networks”. We hope our discussion below sufficiently addresses your remaining questions.

> “It is not clear…what the distribution of the loss change over all samples is.”

For outliers with respect to *change in loss*, we think your idea to consider the entire distribution is a good one! We did visualize this early in our experiments: it is consistently concentrated in a nice bell curve centered near 0. **Per your suggestion, we’ve produced a plot of this distribution for a few iterations and added it to Appendix D (Figure 31).** The mean is indeed very close to 0 (typically  around $-.01$) and the standard deviation is usually around 0.3 to 0.5, sometimes a bit more, but never more than 0.8. The fraction of samples which are more than 2 (or 3 or 4, respectively) sds above/below the mean is consistently around .055 (.015 or .006, respectively). Thus the points which are changing in loss by ~1.0 or more (those depicted in Figure 1) are ~5% or less of the data, which we believe is reasonable to consider an outlier. **Thanks for this idea, we’ll add further discussion of this plot in the main body.**

Reviewer comment (Theory):
The statement of Theorem 3.2 mentions that "sharpness will increase linearly in $\Vert \beta \Vert$ until some time $t_2$". Can an exact formula (or at least more explicit upper/lower bounds) be given for sharpness as a function of time? I believe this should be possible due to having explicit formulae for the weights during training for the linear model. Additionally, is it possible to highlight the role of the problem parameters including $k, \alpha$, and $\Vert \beta \Vert$, and their relationship to the observations for deep networks?

Author response:
Thank you for your thoughtful comments! We are very much in agreement that our finding “can have significant impacts on our understanding of the optimization dynamics of deep neural networks”. We hope our discussion below sufficiently addresses your remaining questions.

> Can an exact formula (or at least more explicit upper/lower bounds) be given for sharpness as a function of time?

Because of the complex interaction between all the relevant terms, deriving bounds as a function of time that are independent of all the other variables is quite difficult. Per the proof of Theorem 3.2, we do at least have that the sharpness is upper bounded by $\mathcal{O}\left( \alpha \exp(||\beta|| t) \right)$ (see Equation 68). In terms of other variables, we can give an upper bound of $\alpha \frac{||\beta||^2}{\epsilon^2}$ (line right after Equation 75), where $\epsilon = \frac{b^\top \beta}{||\beta||}$ is the component of $b$ in the “signal” direction. We also have a matching lower bound (to order) of $\Omega\left(\alpha \frac{||\beta||^2}{\epsilon^2} \right)$. Giving an explicit lower bound for all times $t$ is much harder: our proof works by showing that the minimum satisfies a certain equilibrium, and that the relative growth rates of certain values mean we cannot reach that point without the sharpness exceeding the stated lower bound.

Reviewer comment (Writing):
As a minor comment, the sharpness is typically referred to as the "largest eigenvalue of the loss", while it could be more accurately described as the "largest eigenvalue of the Hessian of the loss".

Author response:
> the sharpness is typically referred to as the "largest eigenvalue of the loss", while it could be more accurately described as the "largest eigenvalue of the Hessian of the loss".

Thanks for pointing out this typo. We have corrected this to “largest eigenvalue of the loss Hessian”.

Reviewer comment (Evaluation):
There is too much informal language and the paper does not consider other confounding factors in leading to the instability. Specifically, the paper uses "per-step largest loss change" to define groups. Of course these groups will have large gradient norm because they have the largest errors in them.

Author response:
> There is too much informal language…the paper uses "per-step largest loss change" to define groups. Of course these groups will have large gradient norm because they have the largest errors in them.

**This is not how we define the groups.** Per-step loss change is how we identify candidate samples which have significant influence—we define the groups via visual inspection (note that if you track our groups throughout training, they do not *always* have large gradient norm). We are very forthright in the first paragraph of Section 2 that these groups are not formally defined but that we feel they are self-evident. When identifying patterns in real-world data, **sometimes the best you can do is to say “we think this is clear, here are some examples so you can judge for yourself”, as we do in Appendix H.** As a thought experiment: suppose instead we were studying a group of CIFAR images that we claim are mislabeled—how could we define this set, other than to visualize them and say “look for yourself”? Clearly, “images whose true label doesn’t match the label in the dataset” would not be a valid definition, since the “true label” cannot be formally defined.

Reviewer comment (Writing):
There are words like genuine, correct, and random noise used but without formal definition. The authors acknowledge this but without clarity on what these are, the proposed phenomenon is too vague. You can define features as the groups of data that the feature helps correctly classify, which becomes a formal definition.

Author response:
> There are words like like genuine, correct, and random noise used but without formal definition. The authors acknowledge this but without clarity on what these are

We agree that it would be ideal to be able to give precise definitions for many of these concepts, beyond the formal definitions we give in our analysis in Section 3.3. Unfortunately, **choosing the "correct" formal definition for real data seems infeasible to us, given our nascent understanding of opposing signals:** e.g., what a “feature” is and whether or not an image contains it is inherently fuzzy (how would one measure what it means to have the “sky” feature, or quantify the “skyness” of a real-world image? Is one blue pixel enough?)

You’ve suggested, for example, defining a “feature” as “the groups of data that the feature helps correctly classify”---we assume you meant here that a feature would be a *property shared by such a group*, rather than the groups themselves. But **this would still not be precise enough:** if planes have a sky background and are also consistently white, are these two separate features? Or just one? What about $x_o$ in our theoretical model (if we consider classification instead of regression), which is certainly a “feature”, but is unable to help a linear classifier separate the two classes and therefore doesn’t fit your definition? Because of this difficulty, outside of theoretical models like the one we study in Section 3.3, it is unclear which of the many possible definitions would be appropriate—especially because our understanding of this phenomenon is so new. Furthermore, we are not aware of any prior work that does this.

We do want to make it as clear as possible that we *intentionally* do not attempt to define “feature” or other necessarily hazy concepts in this work. At several points in the paper (e.g., second paragraph of the intro) we point out that we are presenting our current best understanding only at a high level. **To make this more explicit, we have added a subsection to Section 2 (highlighted in blue) which discusses this point.** We hope this clarifies our intent.

Reviewer comment (Writing):
One issue with the writing is loose language. For example, why call $x_0$ unimportant? Clearly it helps predict the output, just on a subset of data. The idea of opposing signals seems very similar to having a "shortcut". The text in section 3 seems tedious because the authors use language that seems to be intentionally removed from terminology in shortcut learning or distribution shift, but they use an example that is commonly called shortcut learning (model relying on skies to classify planes because many planes have sky in the background).

Author response:
> why call $x_o$ unimportant?

**We believe you may have misread this sentence.** We call the *choice of model distribution* over $x_o$ unimportant, not $x_o$ itself. That is, we are saying that our theoretical results will hold even if it follows a different distribution. We have updated this sentence to make this distinction, please let us know if it is unclear. You are correct that “it helps predict the output, just on a subset of data”, but the problem is that we cannot know a priori which subset a sample belongs to, which means the optimal thing to minimize loss is to not use $x_o$ at all.

> The idea of opposing signals seems very similar to have a "shortcut".

We certainly agree that there is a clear connection between “shortcuts” and opposing signals, and **we point this out in the discussion in Appendix A** (opposing signals are related to many other concepts so the full discussion would not fit in the main body). You are also correct that our running example of “sky” could fit the traditional idea of a shortcut. However, we need to use more generic terminology in Section 3 because **opposing signals and “shortcuts” are not the same.**

As one example, “shortcuts” refer to correlations in the data which allow a classifier to “cheat” by not learning the true decision rule. But one of the opposing signals in Figure 1 is the presence of vehicular features (we can’t define this exactly, but the images all share things like wheels, headlights, windshields, etc.). **These features are in fact the *intended* signal for classifying cars and trucks, so they would not be considered shortcuts.** Furthermore, we describe in Section 3.1 how opposing signals are specifically a way to reduce loss as quickly as possible according to the *current* internal representation, in contrast to “shortcuts” which are fixed statistical relationships of the data which some trained neural networks rely on. So while we agree that there is a clear connection—which we do discuss—there are key differences and it is important to describe opposing signals more generally.

Reviewer comment (Experiments):
The main claim is "opposing signals affect sharpness like so...". There should be a single plot that shows this exactly. I could not find one such experiment. Why not directly measure sharpness, at least for the first few layers? It seems like this was done in "Additional findings", but there is no plot and supporting text. It would be great if there was a plot that showed training instability occurs exactly when sharpness rises up, and then controlling opposing signals via some robust loss or removing some samples both removes the sharpness and training instability.

Author response:
> Why not directly measure sharpness, at least for the first few layers? It seems like this was done in "Additional findings", but there is not plot and supporting text. 

We agree that it would be good to include the figures showing the result we reported in the additional findings. **We have added these figures for all architectures to the updated submission, please see Appendix E.**

> It would be great if there was a plot that showed, training instability occur exactly when sharpness rises up

This is already known to occur. When the sharpness crosses the threshold $\frac{2}{\eta}$, Gradient Descent is no longer guaranteed to reduce the local quadratic Taylor expansion, and instability occurs. See [1] for examples of this and further discussion.

> and then controlling opposing signals via some robust loss or remove some samples both removes the sharpness and training instability.

Figure 2 demonstrates that these individual samples have substantially higher curvature along the top eigenvector than the average point. This *necessarily implies* that removing them would reduce the sharpness. For a similar result regarding robust loss, in Section 3.4 we point to [2], who already show that label smoothing reduces sharpening.

Reviewer comment (Experiments):
The experiments need to be more than a collection of simple post-hoc checks to show that the cause of optimization is certain groups. One direction could be to randomly change the backgrounds of the supposed outlier groups (up to rho fraction) and show the sharpness behavior is avoided.

Author response:
> One direction, maybe is to randomly change the backgrounds of the supposed outlier groups (upto rho fraction) and show the sharpness behavior is avoided.

We are unsure how one could reliably run this experiment. The images are in raw pixel form—how would we “randomly change the backgrounds”? Also, please keep in mind that **opposing signals are not *only* in the image background; we chose this only as a simple running example** (e.g., in Figure 1 two of the signals are “vehicular features” and “grass texture”). This means that changing the background would not necessarily reduce sharpness.

Reviewer comment (Theory):
In EoS, vibrations are observed along the maximal eigen-direction. Could the authors explain how the presence of outliers impacts this phenomenon?

Author response:
Our experiments suggest that the oscillations along the top eigenvector are *caused* by opposing signals. This maximal eigenvector represents **both** (i) the direction with a dominant gradient (see Figure 2; this is because opposing signals have large feature norm); **and** (ii) the direction that will most rapidly imbalance the groups, and thus it has high loss curvature. Item (i) implies that the overall loss gradient is heavily aligned with this top eigenvector—at the same time, because of the curvature described in Item (ii), we will overshoot the minimum, and so the next gradient will point back in the other direction. These two components together cause the phenomenon you’ve noted (we plot this for SGD in Figure 6).

We believe Figure 4 presents convincing evidence of this: during training, the class logits on a sky-colored image bounce back and forth between the “plane” class and a more uniform distribution. As shown in the rightmost subfigure, the feature norm is very large, which causes high curvature [Item (ii)], so the parameters keep overshooting the minimum. And the first two subfigures depict how this oscillation direction (i.e., the gradient we are following) is aligned with the sky feature [Item (i)], causing large changes to the class probabilities.

Reviewer comment (Novelty):
In [1], the authors proposed the concept of multi-view features, which seems to be similar to the Outliers with Opposing Signals in this work. Could the author provide some comparisons?

[1] Allen-Zhu and Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. (ICLR 2023)

Author response:
Thank you for bringing this work to our attention! We expect there are many possible properties of a dataset that could give rise to opposing signals, and multi-view features are one plausible candidate. For example, one could imagine that the multi-view structure could determine probabilities of observing certain combinations of features, which in turn would decide how many samples exhibit a particular feature in each group and how large the gradient is in each of the two directions. Our observations are also consistent with that work’s result about how ensembling improves performance by reducing individual networks’ reliance on particular features, for example if each model learns to downweight opposing signals to different degrees, but no single model eliminates all of them. More fine-grained experimentation on this idea would be an interesting direction for future work.

As we noted in the paper, we believe in many cases our experiments provide strong evidence *in support of* the models in prior work (such as multi-view features), implying that these models realistically capture optimization in deep learning. **We will add this discussion to the related work.**