Paper: Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamics (ICLR-2024)

Reviewer comment (Experiments):
I would anticipate the theoretical framework to align with the behavior observed in real datasets. However, the paper does not investigate the dynamics in more complex scenarios.

Author response:
**More complex scenarios:** Thank you for the suggestion. In the newly added Appendix D.2, we provide the similar experiments to the synthetic case with the ResNet-18 encoder. Overall, we can see a similar trend, and moreover, the regime transition can be observed as well (with properly chosen parameters).

Reviewer comment (Theory):
The authors mentioend BarlowTwins and VICReg. They effectively are still contrastive. How do you think their 'feature normalization' behavior is related?

Author response:
**Barlow Twins and VICReg:** Compared to the conventional contrastive learning pushing anchors away from negative examples, Barlow Twins and VICReg are still different because their variance regularizer is to regularize the _dimension-wise_ variance, not _sample-wise_. Thus, their underlying mechanism to avoid the collapse is still an interesting problem to study, particularly under the presence of feature normalization, which has not been dealt with.

Reviewer comment (Theory):
Some of the assumptions are quite stringent, especially since this paper is not pioneering work, and they may not provide much reference value for practical non-contrastive learning with negative pairs.

Author response:
Thank you for carefully reviewing our manuscript. Let us discuss your concerns about the assumptions. We hope the discussion sounds reasonable and supports our contribution further.

**Assumption of the width-infinite limit:** This assumption is relevant to the norm concentration, which is essentially the central limit theorem. Numerically, the concentration can be established with the order of 100 dimensions; this is not unrealistic limit, unlike some generalization analysis of neural nets requiring exponentially more dimensions than samples.

**Assumption on the norm constancy:** This assumption is only relevant to disentangle the matrix dynamics (4) into the eigenmode dynamics (6). We do not need the norm constancy globally (for all time $t$) if we are concerned about this disentanglement at each fixed time. Globally, the norm values may of course change; however, the shapes of dynamics at each time shown in Figures 2 and 3 remain qualitatively the same.

**General:** Dynamics analysis generally requires strong assumptions such as (isotropic) Gaussian distributions and linear-algebraic assumptions—they are often stronger than practical scenarios. However, this does not mean that these analyses are pointless; as long as they can serve as proxies to some real scenarios. How about our analysis? Isotropic Gaussian—yes, it is quite strong, yet we can establish the regime transition and go beyond the L2-loss analysis (that fails with excessively strong regularization) at least in this specific scenario. The symmetry assumption of $W$ as well: as our simulation actually observes the eigenmodes converge to the stable interval, this can be seen as corroborating evidence that our model behaves as a reasonable proxy. Everyone anticipates the extension to general setups, but not all assumptions can be lifted all at once—our scientific journey is a continuous step to build a better model little by little. To this end, our work contribute to taking feature normalization into account and relating it to complete collapse in a synthetic setup first.

Reviewer comment (Theory):
Assumptions 2 and 3 in section 4 are rather strict. Assumption 2 requires that the input data follow an isotropic Gaussian distribution, which is hard to accept in practical situations. Perhaps a mixture of isotropic Gaussians could be considered. Assumption 3 pertains to the width-infinite limit.

Author response:
Thank you for carefully reviewing our manuscript. Let us discuss your concerns about the assumptions. We hope the discussion sounds reasonable and supports our contribution further.

**Assumption of the width-infinite limit:** This assumption is relevant to the norm concentration, which is essentially the central limit theorem. Numerically, the concentration can be established with the order of 100 dimensions; this is not unrealistic limit, unlike some generalization analysis of neural nets requiring exponentially more dimensions than samples.

**Assumption on the norm constancy:** This assumption is only relevant to disentangle the matrix dynamics (4) into the eigenmode dynamics (6). We do not need the norm constancy globally (for all time $t$) if we are concerned about this disentanglement at each fixed time. Globally, the norm values may of course change; however, the shapes of dynamics at each time shown in Figures 2 and 3 remain qualitatively the same.

**General:** Dynamics analysis generally requires strong assumptions such as (isotropic) Gaussian distributions and linear-algebraic assumptions—they are often stronger than practical scenarios. However, this does not mean that these analyses are pointless; as long as they can serve as proxies to some real scenarios. How about our analysis? Isotropic Gaussian—yes, it is quite strong, yet we can establish the regime transition and go beyond the L2-loss analysis (that fails with excessively strong regularization) at least in this specific scenario. The symmetry assumption of $W$ as well: as our simulation actually observes the eigenmodes converge to the stable interval, this can be seen as corroborating evidence that our model behaves as a reasonable proxy. Everyone anticipates the extension to general setups, but not all assumptions can be lifted all at once—our scientific journey is a continuous step to build a better model little by little. To this end, our work contribute to taking feature normalization into account and relating it to complete collapse in a synthetic setup first.

Reviewer comment (Theory):
In section 5, the authors consider the norms of these linear layers as constants (Assumption 5). This assumption, however, is still far from providing a real dynamic analysis for the cosine loss. Since feature normalization may not guarantee convexity, smoothness, and Lipschitzness, its dynamic analysis should focus on proving the convergence rather than simplifying its complexity to obtain closed-formed dynamics. The existing conclusions do not provide much contribution and insight to understanding non-contrastive learning dynamics.

Author response:
We appreciate the reviewer for clarifying the misunderstanding in my previous post. Regarding Assumption 5, we have the following claim in the initial response:

> **Assumption on the norm constancy:** This assumption is only relevant to disentangle the matrix dynamics (4) into the eigenmode dynamics (6). We do not need the norm constancy globally (for all time) if we are concerned about this disentanglement at each fixed time. Globally, the norm values may of course change; however, the shapes of dynamics at each time shown in Figures 2 and 3 remain qualitatively the same.

To state differently, we _do not_ need the global norm constancy for all time $t$ if we want to characterize the shape of the dynamics at a fixed time $t$; of course, this fixed dynamics cannot be directly tracked along time, but qualitative behaviors of the equilibriums remain the same. We will try to improve the writing of this part to avoid the confusion.

Reviewer comment (Novelty):
There have been recent contributions to the literature of non-contrastive learning which do take into account the cosine loss, and which are not referenced in this article. In particular, Halvagal et al., *Implicit variance regularization in non-contrastive SSL*, 2023. The eigenmode dynamics seem extremely similar (after some changes in the notation) and it seems extremely important that the authors compare themselves to this article. The authors also do not seem to have a similar conclusion on the implicit variance regularization that Halvagal et al. focus on.

Author response:
**Previous literature:** A good catch! It’s a shame that we had been aware of Halvagal et al. (2023) only one week after the submission… They have successfully established the mechanism of the implicit variance regularization of the eigenvalues, which would not be elicitable from our analysis because we assume the standard normal input, regrettably. By contrast, their implicit regularization holds _only for non-collapsed eigenvalues ($\lambda_k > 0$)_, as seen in their Eq. (11). Thus, they do not explain how the complete collapse ($\lambda_k = 0$ for all $k=0$) is avoided; our work contributes to this end. The clarification has been added in the updated draft.

Reviewer comment (Theory):
The authors claim that as the norms decrease, the regimes fall to the stable one. However, in the stable regime, the norms will increase as the eigenmodes increase to the saddle point \(p^+\). Is there a risk of the acute and stable regimes alternating between each other?

Author response:
Regarding the Acute and Stable regimes, we will not have alternation of them numerically because the Stable regime is conceptual and cannot be attained exactly; the unstable interval (gray in Figure 3) could be arbitrarily small as the norms decrease, but will not completely disappear.

Reviewer comment (Experiments):
Numerical experiments on the SimSiam model remain on linear networks in Section 5.4. A similar figure to Figure 6 for a real network such as ResNet (maybe only focusing on the linear projection head) would help confirm the theoretical findings in the linear case. Otherwise, the link with a real SimSiam network remains relatively limited, except for the weight decay argument.

Author response:
**Experiments:**  Thank you for the suggestion. In the newly added Appendix D.2, we provide the similar experiments to the synthetic case with the ResNet-18 encoder. Overall, we can see a similar trend, and moreover, the regime transition can be observed as well (with properly chosen parameters).

Reviewer comment (Presentation):
I find Figure 6.c. hard to read. Are the intervals the theoretical intervals using the values of the norms? In this case, what is the theoretical value of the saddle point? Having the values of a single eigenmode gives very little information on the values of the spectrum of \(W\). Do all the values stay relatively constant like here?

Author response:
**Figure 6c:** We improved the visualization by adding smoothing along the time axis and added a description to explain how the regime intervals are calculated: they are calculated theoretically by using the $p_j$-dynamics (8), but solving the sixth-order equation by numerical root finding. In addition, we expanded the analysis in newly added Appendix D.1 to see not only the largest but also the other eigenvalues. Those eigenvalues converging to non-zero values tend to be in the stable (blue) interval, and all the other converges to zero. More discussion can be found there.

Reviewer comment (Theory):
The notion of Thermodynamic limit is novel to me in optimization and needs to be further explained. How is it different from the Neural Tangent Kernel regime, is it the constant ratio \(\alpha\)?

Author response:
**Thermodynamic limit:** It is relevant to the NTK regime, but the proportional limit with ratio $\alpha$ is important. Such a proportional limit has been often used to study the generalization error under the double descent (for controlling random matrices). For example, Pennington & Worah (2017) “Nonlinear random matrix theory for deep learning” leverages it.

Reviewer comment (Theory):
Do the authors have more intuition on the role of the exponential moving average in BYOL with their new findings?

Author response:
**Findings about EMA:** The most general analysis of non-contrastive learning with the exponential moving average is challenging, so Tian et al. (2021) chose to approximate it by the _proportional EMA_ (their Assumption 1): representing the target representation net by $\Phi_a(t) = \tau(t) \Phi(t)$. While we can do the same analysis with our setup, the proportional ratio $\tau$ is eventually cancelled out due to the feature normalization.

Reviewer comment (Theory):
Considering the majority of the analysis assumes norms of all features are nearly same, due to the high dimensional limit, I do not see how this analysis can show the effects of feature normalization on non-contrastive learning.

Author response:
**Assumption on the norm constancy:** This assumption is only relevant to disentangle the matrix dynamics (4) into the eigenmode dynamics (6). We do not need the norm constancy globally (for all time $t$) if we are concerned about this disentanglement at each fixed time. Globally, the norm values may of course change; however, the shapes of dynamics at each time shown in Figures 2 and 3 remain qualitatively the same.

Reviewer comment (Writing):
Moreover, the notions of "6-th order dynamics" and "3rd order dynamics" are not sufficiently explained in the paper.

Author response:
**6th order vs. 3rd order dynamics:** As can be seen in the dynamics (8) and (9), they are 6th- and 3rd-order formula in $p\_j$, respectively, which are called the dynamics with respective orders. One of the practical differences is admissible hyperparameter ranges; for example, while regularization (weight decay) cannot be too large for the L2 loss, it is possible for the cosine loss, as confirmed in our pilot study (Figure 1).

Reviewer comment (Novelty):
Most importantly, I'm not convinced this an interesting problem to study in the context of prior work providing key understanding regarding how non-contrastive SSL training dynamics work.

Author response:
**Importance of the problem studied here:** Many recent studies have started to theoretically investigate non-contrastive learning and its implicit bias from the perspective of the L2 loss, which may represent different behaviors from the cosine loss as we show in this work. To elicit more useful insights in the future, we believe that our work can definitely contribute to driving the community into focusing on the cosine loss and providing a cornerstone for the analysis.

Reviewer comment (Theory):
Considering the majority of the analysis assumes norms of all features are nearly same, due to the high dimensional limit, I do not see how this analysis can show the effects of feature normalization on non-contrastive learning. Can the authors explain why they believe this analysis is showing anything meaningful about feature normalization.

Author response:
**Assumption on the norm constancy:** This assumption is only relevant to disentangle the matrix dynamics (4) into the eigenmode dynamics (6). We do not need the norm constancy globally (for all time $t$) if we are concerned about this disentanglement at each fixed time. Globally, the norm values may of course change; however, the shapes of dynamics at each time shown in Figures 2 and 3 remain qualitatively the same.

Reviewer comment (Evaluation):
Practically, are there any differences in the conclusions of the training dynamics of the cosine loss and the L2 loss? (while they may be of "different orders").

Author response:
**6th order vs. 3rd order dynamics:** As can be seen in the dynamics (8) and (9), they are 6th- and 3rd-order formula in $p\_j$, respectively, which are called the dynamics with respective orders. One of the practical differences is admissible hyperparameter ranges; for example, while regularization (weight decay) cannot be too large for the L2 loss, it is possible for the cosine loss, as confirmed in our pilot study (Figure 1).