Paper: Zero-Mean Regularized Spectral Contrastive Learning: Implicitly Mitigating Wrong Connections in Positive-Pair Graphs (ICLR-2024)

Reviewer comment (Novelty):
This work appears to be relatively incremental based on Ref. [18] and [19]. Basically, what caught my interest is the representation of spectral embeddings using affinities and features; however, it was previously proposed in [18].

Author response:
Thank you for your kind comment.
At first glance, the formulation of the proposed zero-mean regularized spectral contrastive loss (ZM-SpeCL) may appear similar to the vanilla SpeCL [18] and [19], with the only difference being the introduction of $\tau$ in the term involving negative pair. However, it cannot be simply justified that our work is an incremental improvement over SpeCL. Otherwise, following the same logic, some recent well-known works in self-supervised learning, such as BYOL and SimSiam, could also be considered incremental compared to SimCLR.

More importantly, ZM-SpeCL possesses a clear theoretical motivation and support. It is specifically designed to modify pairwise similarities between positive pairs, which intuitively relaxes the
orthogonality of representations between negative pairs and implicitly alleviates
the adverse effect of wrong connections in the positive-pair graph, leading to better
performance and robustness. Our work is not only simple yet effective, but also theoretically sound.

In Section 3.2, we delve into the role of zero-mean regularization on spectral contrastive pretraining-based unsupervised domain adaptation and provide theoretical proof that zero-mean regularization can tighten the error bound by multiplying a factor less than one. In Section 3.3, we establish the supervised version of spectral contrastive loss and derive the closed-form optimal representations, which resembles the Neural Collapse phenomenon and suggests using class-mean features as a classifier. We further prove that zero-mean regularization can mitigate label noise by implicitly reducing mislabeled weights in the noise transition matrix.

Reviewer comment (Novelty):
Although the authors describe the proposed additive factor as simple yet effective, this work can still be considered progressive research. However, when compared to [18] and [19], it is unclear whether significant progress has been made for this conference.

Author response:
Thank you for your kind comment.
At first glance, the formulation of the proposed zero-mean regularized spectral contrastive loss (ZM-SpeCL) may appear similar to the vanilla SpeCL [18] and [19], with the only difference being the introduction of $\tau$ in the term involving negative pair. However, it cannot be simply justified that our work is an incremental improvement over SpeCL. Otherwise, following the same logic, some recent well-known works in self-supervised learning, such as BYOL and SimSiam, could also be considered incremental compared to SimCLR.

More importantly, ZM-SpeCL possesses a clear theoretical motivation and support. It is specifically designed to modify pairwise similarities between positive pairs, which intuitively relaxes the
orthogonality of representations between negative pairs and implicitly alleviates
the adverse effect of wrong connections in the positive-pair graph, leading to better
performance and robustness. Our work is not only simple yet effective, but also theoretically sound.

In Section 3.2, we delve into the role of zero-mean regularization on spectral contrastive pretraining-based unsupervised domain adaptation and provide theoretical proof that zero-mean regularization can tighten the error bound by multiplying a factor less than one. In Section 3.3, we establish the supervised version of spectral contrastive loss and derive the closed-form optimal representations, which resembles the Neural Collapse phenomenon and suggests using class-mean features as a classifier. We further prove that zero-mean regularization can mitigate label noise by implicitly reducing mislabeled weights in the noise transition matrix.

Reviewer comment (Experiments):
The effectiveness of the proposed method is demonstrated through experiments that involve fewer state-of-the-art methods.

Author response:
Thank you for your constructive comment. In response to your suggestion, we have expanded our experimental evaluation to include comparisons with additional state-of-the-art methods, We have incorporated self-supervised learning methods such as MoCo, SimCLR, and BYOL into our comparative analysis. For these experiments, we employed a PreAct-ResNet18 architecture and conducted training over a total of 200 epochs. The training process utilized the SGD optimizer with a momentum of 0.9, a learning rate of 0.1, and a weight decay of 5e-4. To dynamically adjust the learning rate over the course of the 200 epochs, we implemented the cosine decay learning rate schedule.

The comprehensive experimental results, including comparisons with the aforementioned state-of-the-art methods, are presented in the table below. We believe that this expanded evaluation provides a more thorough understanding of the effectiveness of our proposed method in the context of self-supervised learning.

| Method             | CIFAR-10  | CIFAR-100 | SVHN      |
| :----------------- | :-------- | :-------- | :-------- |
| SimCLR             | 83.84     | 55.04     | 88.07     |
| MoCo               | 73.63     | 51.88     | 82.25     |
| BYOL               | 86.32     | 54.73     | 90.24     |
| SpeCL              | 86.25     | 52.44     | 89.58     |
| SpeCL ($\tau$=0.1) | 86.77     | 53.98     | 90.36     |
| SpeCL ($\tau$=0.2) | 86.93     | 55.25     | 90.80     |
| SpeCL ($\tau$=0.5) | 87.05     | 56.61     | 91.10     |
| SpeCL ($\tau$=1.0) | **87.24** | **56.77** | **91.12** |

Reviewer comment (Writing):
On page 4, there is a missing space in "thatSpeCL".

Author response:
Thank you for your careful review. We have rectified the mentioned typos and addressed some other potential errors.

Reviewer comment (Experiments):
It is suggested to include more benchmark methods in the experiments to avoid categorizing it as an enhanced/improved version of SpeCL method, and instead position it as a robust SSL method.

Author response:
Thank you for your constructive comment. In response to your suggestion, we have expanded our experimental evaluation to include comparisons with additional state-of-the-art methods, We have incorporated self-supervised learning methods such as MoCo, SimCLR, and BYOL into our comparative analysis. For these experiments, we employed a PreAct-ResNet18 architecture and conducted training over a total of 200 epochs. The training process utilized the SGD optimizer with a momentum of 0.9, a learning rate of 0.1, and a weight decay of 5e-4. To dynamically adjust the learning rate over the course of the 200 epochs, we implemented the cosine decay learning rate schedule.

The comprehensive experimental results, including comparisons with the aforementioned state-of-the-art methods, are presented in the table below. We believe that this expanded evaluation provides a more thorough understanding of the effectiveness of our proposed method in the context of self-supervised learning.

| Method             | CIFAR-10  | CIFAR-100 | SVHN      |
| :----------------- | :-------- | :-------- | :-------- |
| SimCLR             | 83.84     | 55.04     | 88.07     |
| MoCo               | 73.63     | 51.88     | 82.25     |
| BYOL               | 86.32     | 54.73     | 90.24     |
| SpeCL              | 86.25     | 52.44     | 89.58     |
| SpeCL ($\tau$=0.1) | 86.77     | 53.98     | 90.36     |
| SpeCL ($\tau$=0.2) | 86.93     | 55.25     | 90.80     |
| SpeCL ($\tau$=0.5) | 87.05     | 56.61     | 91.10     |
| SpeCL ($\tau$=1.0) | **87.24** | **56.77** | **91.12** |

Reviewer comment (Experiments):
In the experiments, it is recommended to analyze the results achieved with different values of $\tau$ due to its previous detailed analysis. This analysis can provide insights into what can be inferred from the results obtained with different $\tau$ values.

Author response:
We appreciate your valuable feedback and would like to provide additional insights into the role of the regularization parameter $\tau$ and the trade-offs associated with its selection. In this paper, we propose setting $\tau\in[0,1]$ to balance the regularization strength of zero-mean regularization. Although in Section 3, we do not explicitly constrain $\tau\le 1$, theoretical and empirical evidence supports the necessity of such a constraint:


- In Theorem 3.1, we assume that $\tau<\frac{\tilde{\lambda}_1(0)-\tilde{\lambda}_d(0)}{\tilde{\lambda}_1(0)}<1$ to ensure that the introduction of $\tau$ will not alter the value of the $d$-th eigenvalue in Eq. (C.23). In Proposition 3.2, we assume that $\tau\le \frac{\tilde{\lambda}_1(0)-\tilde{\lambda}_2(0)}{\tilde{\lambda}_1(0)}< \frac{\tilde{\lambda}_1(0)-\tilde{\lambda}_d(0)}{\tilde{\lambda}_1(0)}<1$ to guarantee that the first eigenvalue of $\tilde{A}'$ is $\tilde{\lambda}_1(\tau)=(1-\tau)\tilde{\lambda}_1(0)$ (as can be seen, $\tau\le 1$ is essential to avoid the presence of negative eigenvalues of $\tilde{A}'$) in Eq. (C.32), thus facilitating a more intuitive comparison with the bound at $\tau=0$. While it is possible to draw conclusions within the range of $[\frac{\tilde{\lambda}_1(0)-\tilde{\lambda}_d(0)}{\tilde{\lambda}_1(0)}, 1]$, it requires a more nuanced comparison of the order of these eigenvalues, predictably resulting in an obviously different bound than at $\tau=0$.

- In Theorem 3.3, we prove that the global minimum of the supervised spectral contrastive loss is achieved at $\hat{H}=\arg\min_{H}\|(rI-\tau11^\top)-H^\top H\|_F^2$. When $r\le d+1$, $\|(rI-\tau11^\top)-H^\top H\|_F^2$ has a minimum of zero only for $\tau\in [0,1]$, corresponding to $\hat{H}^\top \hat{H}=rI-\tau11^\top$, where the angle between representations of different classes is $\arccos\frac{-\tau}{r-\tau}>90^\circ$. However, for $\tau > 1$, there exists no $\hat{H}$ that satisfies $\hat{H}^\top \hat{H}=rI-\tau11^\top$, as it would lead to a paradoxical situation where $0\le 1^\top\hat{H}^\top \hat{H} 1= 1^\top(rI-\tau 11^\top)1=r^2(1-\tau)<0$.

- We conducted experiments for $\tau>1$ as presented in Tables 9 and 11 of the Appendix. As observed, consistently better results are obtained with $\tau\in[0,1]$, while excessively large $\tau$ values hinder overall performance. Thus, we suggest limiting $\tau$ to the range of $[0,1]$ to balance the regularization and discriminative capabilities.

Many thanks to the reviewer for the valuable suggestion, we have added this part to Section A.4 in the revised version.

Reviewer comment (Novelty):
The novelty is an issue. According to the Equ 3.2, it is the same as contrastive laplacian eigenmaps (NeurIPS 2021). In contrastive laplacian eigenmaps, they have the same three terms. The main point is the fully connected adjacent matrix (the all-one matrix).

Author response:
Thank you for your insightful comment. While there are certain similarities between Zero-mean regularized spectral loss (Zero-SpeCL) in Eq. 3.2 and contrastive Laplacian eigenmaps (COLES) [1], there are distinct differences between them:

- `Different Negative Components.` The contrastive objective (to be maximized) can generally be formulated as $J(f)=E_{(x,x^+)\sim p_{pos}} s(f(x),f(x^+)) + \eta E_{(x,x^-)\sim p_{data}} \tilde{s}(f(x),f(x^-))$. While the positive components of Zero-SpeCL and COLES are the same, their negative components are different. The negative component in COLES is $\tilde{s}(f(x),f(x^-))=-f(x)^\top f(x^-)$, whereas in Zero-SpeCL, it is $\tilde{s}(f(x),f(x^-))=-(f(x)^\top f(x^-)+\tau)^2$.
- `Different Reasons for Covariance Term.` Both Zero-SpeCL and COLES introduce the term $\Vert \mathbb{E}_x f(x) f(x)^\top-I\Vert_F^2$. The motivation of COLES is to softly satisfy the constraint $F^\top F=I$ that removes an arbitrary scaling factor in the embeddings [2]. Additionally, it helps avoid collapsed solutions since, without the constraint $F^\top F=I$, the derived graph Dirichlet energy $-Tr(F^\top\Delta W F)$ would be minimized when all representations collapse to a constant vector. In contrast, the covariance term $\mathcal{R}_1(f)=\Vert \mathbb{E}_x f(x) f(x)^\top-I\Vert_F^2$ in Zero-SpeCL appears as a part of an equivalent form in Eq. 3.2.
- `Different Overall Objective.` COLES represents a constrained graph Dirichlet energy minimization problem (contrastive Laplacian eigenmaps), i.e.,  $\min_{F^\top F=I}-Tr(F^\top\Delta W F)$. On the other hand, Zero-SpeCL depicts the low-rank matrix approximation problem $\min_F|A-F^\top F|_F^2$ (as shown in Eq. 3.3).
- `Different Focuses.` Though both works revolve around contrastive learning, our work pays more attention on discussing in theory how zero-mean regularization is helpful in downstream unsupervised domain adaptation (UDA) and supervised classification with noisy labels. We theoretically prove that zero-mean regularization can tighten the error bound by multiplying a factor less than one for UDA (Section 3.2), and prove that zero-mean regularization can mitigate label noise by implicitly reducing mislabeled weights in the noise transition matrix (Section 3.3).

Thanks again to the reviewers for their comments. In order to better clarify our contribution, we have added the relevant comparison to contrastive laplacian eigenmaps in the revised version.

>[1] Zhu H, Sun K, Koniusz P. Contrastive laplacian eigenmaps[J]. Advances in Neural Information Processing Systems, 2021, 34: 5682-5695.
>
> [2] Belkin M, Niyogi P. Laplacian eigenmaps and spectral techniques for embedding and clustering[J]. Advances in neural information processing systems, 2001, 14.

Reviewer comment (Novelty):
I would like to hear from authors about the relationship between this one and contrastive laplacian eigenmaps. I think the authors give a totally different theoretical analysis based on a very similar thing.

Author response:
Thank you for your insightful comment. While there are certain similarities between Zero-mean regularized spectral loss (Zero-SpeCL) in Eq. 3.2 and contrastive Laplacian eigenmaps (COLES) [1], there are distinct differences between them:

- `Different Negative Components.` The contrastive objective (to be maximized) can generally be formulated as $J(f)=E_{(x,x^+)\sim p_{pos}} s(f(x),f(x^+)) + \eta E_{(x,x^-)\sim p_{data}} \tilde{s}(f(x),f(x^-))$. While the positive components of Zero-SpeCL and COLES are the same, their negative components are different. The negative component in COLES is $\tilde{s}(f(x),f(x^-))=-f(x)^\top f(x^-)$, whereas in Zero-SpeCL, it is $\tilde{s}(f(x),f(x^-))=-(f(x)^\top f(x^-)+\tau)^2$.
- `Different Reasons for Covariance Term.` Both Zero-SpeCL and COLES introduce the term $\Vert \mathbb{E}_x f(x) f(x)^\top-I\Vert_F^2$. The motivation of COLES is to softly satisfy the constraint $F^\top F=I$ that removes an arbitrary scaling factor in the embeddings [2]. Additionally, it helps avoid collapsed solutions since, without the constraint $F^\top F=I$, the derived graph Dirichlet energy $-Tr(F^\top\Delta W F)$ would be minimized when all representations collapse to a constant vector. In contrast, the covariance term $\mathcal{R}_1(f)=\Vert \mathbb{E}_x f(x) f(x)^\top-I\Vert_F^2$ in Zero-SpeCL appears as a part of an equivalent form in Eq. 3.2.
- `Different Overall Objective.` COLES represents a constrained graph Dirichlet energy minimization problem (contrastive Laplacian eigenmaps), i.e.,  $\min_{F^\top F=I}-Tr(F^\top\Delta W F)$. On the other hand, Zero-SpeCL depicts the low-rank matrix approximation problem $\min_F|A-F^\top F|_F^2$ (as shown in Eq. 3.3).
- `Different Focuses.` Though both works revolve around contrastive learning, our work pays more attention on discussing in theory how zero-mean regularization is helpful in downstream unsupervised domain adaptation (UDA) and supervised classification with noisy labels. We theoretically prove that zero-mean regularization can tighten the error bound by multiplying a factor less than one for UDA (Section 3.2), and prove that zero-mean regularization can mitigate label noise by implicitly reducing mislabeled weights in the noise transition matrix (Section 3.3).

Thanks again to the reviewers for their comments. In order to better clarify our contribution, we have added the relevant comparison to contrastive laplacian eigenmaps in the revised version.

>[1] Zhu H, Sun K, Koniusz P. Contrastive laplacian eigenmaps[J]. Advances in Neural Information Processing Systems, 2021, 34: 5682-5695.
>
> [2] Belkin M, Niyogi P. Laplacian eigenmaps and spectral techniques for embedding and clustering[J]. Advances in neural information processing systems, 2001, 14.