Paper: Modulate Your Spectrum in Self-Supervised Learning (ICLR-2024)

Reviewer comment (Experiments):
The authors seem to have a missing baseline [1] in SSL. The baseline looks similar to INTL from the viewpoint of spectral adjusting. [1] Exploring the Gap between Collapsed & Whitened Features in SSL-ICML2022

Author response:
Thank you sincerely for your valuable suggestions. The theoretical contribution of [1] has been discussed in the related work of our submission, here we further list the key distinctions between our work and [1] as follows.
1. The fundamental concept in [1] revolves around the impact of feature whitening on generalization, with an emphasis on adjusting the whitening degree of pre-trained SSL models based on the power law behavior of eigenspectra to **enhance downstream performance**. In contrast, our paper seeks to introduce a novel SSL training algorithm, leveraging our ST framework, to **address dimensional collapse** without relying solely on whitening.
2. Our paper focuses on analyzing how adjusting the spectrum during the **training phase** can mitigate collapse-related issues. Conversely, [1] primarily investigates how the feature spectrum influences generalization during the **evaluation stage**.
3. The method proposed in [1], named PMP, serves as a **post-processing evaluation technique** aimed at adjusting the spectrum of a pre-trained SSL encoder for improved downstream performance. In contrast, our proposed method, INTL, aligns with **SSL training algorithms** like SimCLR, Barlow Twins, etc. It is designed to prevent collapse within joint embedding architectures during the training process.

[1] demonstrates that the proposed PMP method is effective in enhancing the linear evaluation performance of encoders pre-trained by SimCLR and Barlow Twins, particularly on datasets with limited labeled samples. In line with your recommendations, we conducted experiments on 1% and 10% subsets of ImageNet to assess the performance of our INTL using PMP. The results, presented in the table below, include LP (standard linear probe) and PMP (the evaluation method proposed in [1]). The findings indicate that PMP contributes to the improved evaluation performance of INTL.   
&emsp; However, it's essential to note that our experiments in Section 5 are conducted based on the standard SSL benchmark using a standard linear probe, and therefore, the baseline from [1] is not incorporated in our experimental setup.

|  Method   | Eval | Top-1 | Top-5  |
| :-------- | :--------:| :--------:| :--------:| 
|    |  |1% &nbsp; 10%| 1% &nbsp; 10%|
| SimCLR  | LP |48.1  &nbsp;   61.0 |  73.8   &nbsp;  84.3|
| SimCLR  | PMP |50.9   &nbsp;  62.5 | 76.6  &nbsp;   85.2  |
 |  |  |  | 
| INTL (ours) | LP  | 53.9   &nbsp;  66.4 | 79.1 &nbsp;   87.7  | 
| INTL (ours) | PMP  | 54.9  &nbsp;   67.8 | 79.6   &nbsp;   89.1  |

Reviewer comment (Experiments):
Could the authors compare the INTL method with the baseline [1] if possible?

Author response:
Thank you sincerely for your valuable suggestions. The theoretical contribution of [1] has been discussed in the related work of our submission, here we further list the key distinctions between our work and [1] as follows.
1. The fundamental concept in [1] revolves around the impact of feature whitening on generalization, with an emphasis on adjusting the whitening degree of pre-trained SSL models based on the power law behavior of eigenspectra to **enhance downstream performance**. In contrast, our paper seeks to introduce a novel SSL training algorithm, leveraging our ST framework, to **address dimensional collapse** without relying solely on whitening.
2. Our paper focuses on analyzing how adjusting the spectrum during the **training phase** can mitigate collapse-related issues. Conversely, [1] primarily investigates how the feature spectrum influences generalization during the **evaluation stage**.
3. The method proposed in [1], named PMP, serves as a **post-processing evaluation technique** aimed at adjusting the spectrum of a pre-trained SSL encoder for improved downstream performance. In contrast, our proposed method, INTL, aligns with **SSL training algorithms** like SimCLR, Barlow Twins, etc. It is designed to prevent collapse within joint embedding architectures during the training process.

[1] demonstrates that the proposed PMP method is effective in enhancing the linear evaluation performance of encoders pre-trained by SimCLR and Barlow Twins, particularly on datasets with limited labeled samples. In line with your recommendations, we conducted experiments on 1% and 10% subsets of ImageNet to assess the performance of our INTL using PMP. The results, presented in the table below, include LP (standard linear probe) and PMP (the evaluation method proposed in [1]). The findings indicate that PMP contributes to the improved evaluation performance of INTL.   
&emsp; However, it's essential to note that our experiments in Section 5 are conducted based on the standard SSL benchmark using a standard linear probe, and therefore, the baseline from [1] is not incorporated in our experimental setup.

|  Method   | Eval | Top-1 | Top-5  |
| :-------- | :--------:| :--------:| :--------:| 
|    |  |1% &nbsp; 10%| 1% &nbsp; 10%|
| SimCLR  | LP |48.1  &nbsp;   61.0 |  73.8   &nbsp;  84.3|
| SimCLR  | PMP |50.9   &nbsp;  62.5 | 76.6  &nbsp;   85.2  |
 |  |  |  | 
| INTL (ours) | LP  | 53.9   &nbsp;  66.4 | 79.1 &nbsp;   87.7  | 
| INTL (ours) | PMP  | 54.9  &nbsp;   67.8 | 79.6   &nbsp;   89.1  |

Reviewer comment (Experiments):
As analyzed in this paper, both whitening methods and INTL are instances of spectral transformations. However, it seems that INTL outperforms whitening in every task. So what are the disadvantages of whitening methods? It would be better to provide more theoretical and empirical comparisons between them.

Author response:
Following your constructive suggestions, we further analysis the disadvantages of whitening methods and explain why our INTL can outperform whitening methods in downstream tasks.
1. **The disadvantages of whitening methods.**
>&emsp; In addition to the numerical instability issues discussed in Appendix D.3 for existing whitening methods, the unsatisfactory performance of these methods can be attributed to their weak constraints on embedding which are challenging to control and enhance effectively.  
>&emsp; Recent theoretical work [1] proposes a power law to analyze the relationship between eigenspectrum and representation quality. Meanwhile, [2] demonstrates that the degree of feature whitening influences generalization. Both of these works emphasize that **a steep spectrum diminishes representation quality, a smooth spectrum compromises generalization, while a moderate spectrum ensures optimal generalization ability**. SSL methods impose their constraints on embedding to prevent collapse, and the strength of these constraints directly influences the spectrum distribution.  
>&emsp; Hard whitening methods, such as W-MSE and CW-RGP, impose a full-rank constraint on embedding [3]. However, [3] indicates that the full-rank constraint is too weak to achieve optimal performance, leading to the proposal of a random group partition technique to enhance constraints and improve results. Nevertheless, this enhancement is unpredictable and comes with additional computational costs, especially on large datasets.  
>&emsp; Soft whitening methods, like Barlow Twins and VICReg, impose a strong whitening constraint on embedding, urging the covariance matrix of the embedding to be identity. The results in Table 3 show that these methods (Barlow Twins) perform even worse than hard whitening methods (W-MSE) in transfer learning, which indicates that excessive constraints have already compromised their generalization ability for downstream tasks.

2. **Why our INTL can outperform whitening methods in downstream tasks**
>&emsp; The commendable performance of INTL can be attributed to its moderate constraint on the embedding. Unlike whitening methods, our INTL provides a moderate equal-eigenvalues constraint on embedding, which is stronger than hard whitening (full-rank constraint) but weaker than soft whitening (whitening constraint) as illustrated in Section 4. This moderate constraint results in a satisfactory spectrum distribution, thereby endowing the representation with better generalization capabilities.

Reviewer comment (Theory):
The motivation behind the trace loss is a little confusing. Is it possible to provide a more detailed discussion?

Author response:
As illustrated in Theorem 1, IterNorm implicitly maps  $\\forall \\lambda\_i \\in \\lambda(\\mathbf{Z})$ to $\\widehat{\\lambda}\_i = \\frac{\\lambda\_i}{tr(\\Sigma)} {f\_T}^2(\\frac{\\lambda\_i}{tr(\\Sigma)})$. Based on Formula 4 and 5, we know  $\\forall T\\in \\mathbb{N}, \\lambda\_i > 0 \\Longrightarrow 0 < \\widehat{\\lambda}\_i < 1 $, and $\\lim\_{T\\to\\infty} \\widehat{\\lambda}\_i = 1 $. This suggests that the eigenvalues of the covariance matrix $\\Sigma\_{\\widehat{\\mathbf{Z}}}$ resulting from the transformation output by IterNorm all fall within the range of 0 to 1, with an ideal convergence to 1 in the limit of infinite iterations ($T$). Therefore, $\\forall T\\in \\mathbb{N}, trace(\\Sigma\_{\\widehat{\\mathbf{Z}}})=\\sum\\limits\_{i=1}^d \\widehat{\\lambda}\_i < d$.  
&emsp; Consequently, in experiments, we have noted that IterNorm encounters severe dimensional collapse and struggles to effectively train the model in self-supervised learning, regardless of the chosen value for T. In such instances, we observe some eigenvalues of the covariance matrix $\\Sigma\_{\\widehat{\\mathbf{Z}}}$ collapse into 0, causing a substantial gap between $trace(\\Sigma\_{\\widehat{\\mathbf{Z}}})$ and $d$.  

&emsp; To address this issue, the motivation behind the trace loss is to introduce an additional penalty that encourages  $trace(\\Sigma\_{\\widehat{\\mathbf{Z}}})$ to approach its ideal maximum, where $\\sum\\limits\_{i=1}^d \\widehat{\\lambda}\_i = d$ and $\\forall i, \\widehat{\\lambda}\_i =1$ . By doing so, the collapse is mitigated since it becomes improbable for eigenvalues of embedding to be 0.

Reviewer comment (Evaluation):
It seems that INTL shows superior performance in 5-nn accuracy than linear probing accuracy. Are there any intuitive explanations for that?

Author response:
Thanks for your inspiring questions! Our intuitive explanation is as follows:  
&emsp; The accuracy of the k-nn classifier indicates the state of the sample distribution within the representation's local structure. The k-nn classifier tends to exhibit better classification performance under the condition that samples of the same class are closer in the feature space and those of different classes are farther apart. However, under the absence of data labels in SSL, ensuring this condition can be challenging. For instance, although contrastive learning aims to pull positive pairs together and push negative pairs apart from positive one, the negative pairs may belong to the same class as positive one, in which samples of the same class are erroneously pulled apart. This situation can lead to a decrease in the accuracy of the k-nn classifier. As shown in Table 1, the contrastive method SimCLR performs poorly when using k-nn classifier.  
&emsp; Meanwhile, this situation can also occur in non-contrastive learning as illustrated in [2]. The strength of method constraints on embedding can influence sample distribution. our INTL introduces a moderate equal-eigenvalues constraint on the embedding. We conject that this moderate constraint on embedding results in a tendency for a well-balanced similarity among different samples within the dataset: samples from distinct categories can be effectively distinguished, and samples from the same category are not heavily pushed apart. This equilibrium in the local structure of INTL's representation provides effective guidance for sample classification, contributing to its superior performance in 5-nn accuracy.

Reviewer comment (Writing):
There are some typos. For example, in p.4, ‘Eqn. 13 can be viewed as an optimization problem over …’ should be replaced with ‘Eqn.6 …’.

Author response:
We thank the reviewer and have fixed all these in the revised manuscript.