From: ICLR.cc/2023/Conference/Paper2523/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper2523/Authors
title: A Summary of Paper Updates 
comment: We thank all the reviewers for their constructive suggestions and criticism. 
For most of the reviewers' concern, we modify sec 4.2 and add appendix E to provide clarification to our theory on why SimSiam also minimize the cross-correlation matrix loss.

From: ICLR.cc/2023/Conference/Paper2523/Reviewer_7MYi
title: Thank you for the response
comment: Thank you for the clarifications.
Regarding the assumptions, I think [1] has the same problem of the assumptions being too strong, and [2] is different in that its clustering assumption is on the input data and it shows that the learned representations can recover such clustering structure, rather than directly assuming that the learned representations are clustered.
Therefore, I still think the assumptions are too strong and would like to keep my original score.

From: ICLR.cc/2023/Conference/Paper2523/Authors
title: Thanks for your review. Here we provide discussions on some of your concern:1)Questionable assumptions 2)Writing
comment: 1. For your concern that the 2 assumptions are too strong: 
- For assumption 1, the assumption does not completly apply to the real world data due to limited sampling. However, we test the average $\frac{\|f( \hat{x}_1)-f( \hat{x}_2)\|}{r}$ to estimate $\varepsilon$ with outcome less than 0.1 for both models, which can be acceptable.
- For assumption 2, When the data augmentation make views of two distinct samples very close in the input space(e.g. two different categories get cropped out except for the background), it is very likely that the latent class of the augmented view does not and should not remain the same as the latent class of the original data(background for augmented views differ from original categories), which lead to classification error. The error introduced by data augmentation changing the latent class of the data can be isolated from our analysis. We will work on lessening the non-overlapping assumption.
- In addition, other study on generalization of contrastive learning [1] also use the same assumption, [2] use the continuous assumption as we do .

2. In the original manuscript, why SimSiam also minimize the cross-correlation matrix loss was not explicitly analyzed. We will provide a detailed analysis in our revised manuscript to show how it is done. Please refer to sec 4.2 and appendix E. We will also rectify other writing problem.

[1] Weiran Huang, Mingyang Yi, and Xuyang Zhao. Towards the generalization of contrastive self-supervised learning. arXiv preprint arXiv:2111.00743, 2021.

[2] Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in neural information processing systems, 34:5000–5011, 2021.

From: ICLR.cc/2023/Conference/Paper2523/Authors
title: Thanks for your review. Here we provide discussions on some of your concern:1) Strong assumptions 2) Theoretical analysis of SimSiam 3) Writing and others
comment: 1. For your concern that the Lipschitz continuous assumptions are too strong: 
The assumption does not completly apply to the real world data due to limited sampling. However, we test the average $\frac{\|f( \hat{x}_1)-f( \hat{x}_2)\|}{r}$ to estimate $\varepsilon$ with outcome less than 0.1 for both models, which can be acceptable. In addition, other study on generalization of contrastive learning [1] also use the same assumption, which does not influence the intrinsic value of our theory, [2] use the continuous assumption as we do.

2. We provide a detailed analysis and correlation matrix visualization in our revised manuscript to show why SimSiam also minimize the cross-correlation matrix loss. Please refer to sec 4.2 and appendix E. 

3. Formerly we regard latent class as a common sense, which means the unobserved subgroups. We will try conducting further experiments on ImageNet dataset to explore how the theory can guide model selection.

[1] Towards the generalization of contrastive self-supervised learning. Paper 1647 of ICLR 2023 Conference Blind Submission
[2] Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in neural information processing systems, 34:5000–5011, 2021.

From: ICLR.cc/2023/Conference/Paper2523/Authors
title: Thanks for your review. Here we provide discussions on some of your concern:1)Strong assumptions 2) Theoretical analysis of SimSiam 3) Writing and others
comment: 1. For your concern that the non-overlapping data augmentation assumption are too strong: 
When the data augmentation make views of two distinct samples very close in the input space(e.g.  two different outdoor categories get cropped out except for sky), it is very likely that the latent class of the augmented view does not and should not remain the same as the latent class of the original data(sky for augmented views differ from original categories), which lead to classification error. The error introduced by data augmentation changing the latent class of the data can be isolated from our analysis. We will work on lessening the non-overlapping assumption. In addition, other study on generalization of contrastive learning [1] also use the same assumption as we do.

2. For you concern on potentially inconsistent theoretical assumptions and imprecise theoretical argument:
In Eqn. 16-17 of our original manuscript, the conclusion that $W_p$ is a diagonal matrix can be obtained without isotropic assumptions on data (see appendix D in Tian et al (2021)) 
We will provide a new detailed analysis in our revised manuscript excluding the usage of $\sigma$ to show why SimSiam also minimize the cross-correlation matrix loss. Please refer to sec4.2 and appendix E. 

3. $\sigma$  and $\eta$  in Eqn. 16-17 of our original manuscript are defined in the following as variance of augmented view and the weight decay parameter. In Eqn. 1, $(f(x_1)f(x_2))$ does mean inner product and should be changed into $f(x_1)^Tf(x_2)$ to make it more explicit. $\|A\|$ means Frobenius norm. We have made them clarified in our revised manuscript and we will improve symbols definition in future release.

[1] Weiran Huang, Mingyang Yi, and Xuyang Zhao. Towards the generalization of contrastive self-supervised learning. arXiv preprint arXiv:2111.00743, 2021.

From: ICLR.cc/2023/Conference/Paper2523/Authors
title: Thanks for your review. Here we provide discussions on some of your concern:1) Experiment details 2) Theoretical analysis of SimSiam
comment: 1. For experiment details, we follow the default hyperparameters and optimization method for both Barlow Twins and Simsiam model as the public codebase. For data augmentation we disable GaussianBlur for both model. Following [1], we changed the number of class, filter size, stride, and padding in the Resnet-18 backbone so that it works with CIFAR dataset.
We will try conducting further experiments on ImageNet dataset to explore how the theory can guide model selection.

2. We will provide a detailed analysis in our revised manuscript to show why SimSiam also minimize the cross-correlation matrix loss. Please refer to sec 4.2 and appendix E. 

[1]https://github.com/huyvnphan/PyTorch_CIFAR10

From: ICLR.cc/2023/Conference/Paper2523/Authors
title: Thanks for your review. Here we provide discussions on some of your concern:1)Strong assumptions 2) Loss analysis 3)Experiment details
comment: 1. For your concern that the 2 assumptions are too strong: 
- For assumption 1, the assumption does not completly apply to the real world data due to limited sampling. However, we test the average $\frac{\|f( \hat{x}_1)-f( \hat{x}_2)\|}{r}$ to estimate $\varepsilon$ with outcome less than 0.1 for both models, which can be acceptable.
- For assumption 2, when the data augmentation changes the latent class of the data, it is very likely that the latent class of the augmented view does not and should not remain the same as the latent class of the original data, which lead to classification error. The error introduced by data augmentation changing the latent class of the data can be isolated from our analysis. We will work on lessening the non-overlapping assumption.
- In addition, other study on generalization of contrastive learning [1] also use the same assumption, [2] use the continuous assumption as we do .

2. In theoretical analysis, Barlow Twins explicitly minimize the cross-correlation matrix loss, whether the output features normalized or not, so we use the original Barlow Twins model. We will provide a detailed analysis in our revised manuscript to show why SimSiam also minimize the cross-correlation matrix loss. Please refer to sec4.2 and appendix E. 

3. In the experiment, downstream task (image classification) is comprised of these latent classes. In the testing phase, target labels are introduced to define the number of the latent class and predicted label were calculated to measure the test accuracy. Fig 4: and Table 1 are the accuracies on test batches by single runs. We will perform multiple runs in future release.

[1] Weiran Huang, Mingyang Yi, and Xuyang Zhao. Towards the generalization of contrastive self-supervised learning. arXiv preprint arXiv:2111.00743, 2021.

[2] Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. Advances in neural information processing systems, 34:5000–5011, 2021.

From: ICLR.cc/2023/Conference/Paper2523/Reviewer_7MYi
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper studies the generalization performance of a nearest neighbor classifier.

The upper bound for generalization error involves a term $\|\Lambda - I_d\|_F^2$, where $\Lambda$ is the covariance matrix.
Hence minimizing this term tightens the generalization bound.

The paper shows that two non-contrastive method both minimizes this term:
- Barlow Twins (with $\lambda=1$) explicitly minimizes this term;
- SimSiam implicitly minimizes this term by having the same eigenvalue for non-collapsing dimensions.

The paper then empirically verifies that the generalization upper bound correlates well with the kNN accuracy, for Barlow Twins and SimSiam on CIFAR10 and CIFAR100.
strength_and_weaknesses: On the plus side, the paper proposes a quantity $\mathcal{L}_{bound}$, which is used as the generalization bound and can be calculated from unlabeled data.
$\mathcal{L}\_{bound}$ empirically correlates well with the kNN accuracy.
The paper is also clearly structured and easy to follow.

However, I'm concerned about the significant of both the theoretical and the empirical results.

Notes for theoretical results:
- For the generalization bound, the idea is that if 1) the encodings from the same class are close to each other and that 2) there is no overlapping between classes, then the error on the kNN classifier is bounded; the proof then directly follows from algebraic manipulations.
My concern is that these 2 assumptions are too strong, since they imply that the embeddings are already well-clustered, hence a kNN classifier is expected to work well.
  - Minor comment: Thm 2: $p_i$ is the probability of class $i$; the definition can be simplified since $\mathbf{1}[\hat{x} \in \mathcal{A}(S_i)]$ iff $x \sim \mathcal{C}_i$ by the assumption that classes are non-overlapping.
- For Barlow Twins, the connection to minimizing $\|\Lambda - I_d\|_F^2$ is shown only for $\lambda = 1$. The derivation also does not consider the fact that $C$ is calculated on normalized features, not the direct encoder output.
- For SimSiam, the connection to minimizing $\|\Lambda - I_d\|_F^2$ relies on eigenspace alignment results in prior work (Tian et al. 2021), and is shown only for taking time $t \rightarrow \infty$ on non-collapsing dimensions; no comment is provided for whether and why the non-collapsing solution is preferred over the collapsing solution.

Notes for empirical results:
- If the number of class $K$ is unknown, can we estimate $K$ from the data? How would the relation between $\mathcal{L}_{bound}$ (with the estimated $K$) and test accuracy be?
- Fig 4: are the accuracies on test batches or training batches?
- Table 1 & Figure 4: it seems that each number is obtained by a single run, based on the zigzag behavior in Fig 4. Please consider showing the results from multiple runs (i.e. multiple random seeds) to reduce the effect of randomness.


clarity,_quality,_novelty_and_reproducibility: _Writing_: the paper is clear and easy to follow. Some comments:
- There are unnecessary details on contrastive methods (SimCLR) since it's not used in any later part of the paper. For example, Fig 1(a) and equation (1) can be removed.
- There should be more related work on non-contrastive methods; currently both subsections are titled with _contrastive_ learning. Some missing references to consider include https://arxiv.org/abs/2110.04947 and https://arxiv.org/abs/2203.15702.
- In last paragraph of the related work, about improving generalization performance by changing the loss function: please add citations.

_Novelty_: I find the novelty to be limited for both theory and experiments.

_Reproducibility_: experiment details could be better clarified.
summary_of_the_review: This paper upper bounds the generalization error of a kNN classifier with a term $\|\Lambda - I_d\|_F^2$, and empirically verifies the correlation.

While the writing is clear, I find the contributions insufficient and would recommend a reject.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper2523/Reviewer_gMu2
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper studies non-contrastive learning methods such as Barlow-Twins and SimSiam. It propose a theoretical guarantee on their clustering ability.
strength_and_weaknesses: This paper has a serious drawback: its theorem is basically a direct corollary of its assumptions. In Sec 4.1, the paper assumes 
1. two data points in the same class have very similar representations, characterized by $||f(x) - f(x')||_2\leq ||f(x)||_2^2 \epsilon$, while different latent classes have no intersection. 
2. the representation map $f$ is Lipschitz continuous.

Under these assumptions, it is fairly easy to obtain $\epsilon^2 + loss/||f(x)||^2$ error of the downstream latent class classification. Such a result does not provide any new perspective for understanding these non-contrastive methods, nor does it help in providing a new theoretical framework for characterizing its behaviors.
clarity,_quality,_novelty_and_reproducibility: Most of the paper is not written clearly. The assumptions of the data are not put in an definition or assumption environment and hard to find, and the theorem statements lack sufficient description. Moreover, Sec 4.2 is almost unreadable and incomprehensible, with results and equations borrowed from previous work without careful explanation.
summary_of_the_review: This paper studies an important problem of providing generalization guarantees for non-contrastive learning. However, this paper has limited contribution toward this research direction as their assumptions are too strong and no novel message is conveyed. Moreover, the writing of this paper is rather horrible.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 1: The contributions are neither significant nor novel.
flag_for_ethics_review: NO.
recommendation: 1: strong reject

From: ICLR.cc/2023/Conference/Paper2523/Reviewer_w6HN
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper studies the generalization of Barlow Twins and SimSiam. The authors focus on the generalization error of the inter-class. This generalization bound is determined by the embedding dimension. The authors conducted an empirical study on the experiments on CIFAR-10 and CIFAR-100 datasets using kNN. The overall accuracies approximately fit the predicted generalization bound.
strength_and_weaknesses: Strengths:
1. The authors first linked the non-contrastive inter-class distance to the classification generalization bound. The theory is simple and correct.
2. The experiment results approximately fits the theory's prediction. This gives insight into how to choose embedding dimensions and possibly understand dimensional collapse.


Weaknesses:
1. The experiments are conducted on a small scale and only using kNN. It is questionable how each dimension setting is optimized. Do they follow the same hyperparameters, or is each one tuned? The authors mentioned that they are using the public codebase. But these are optimized for ImageNet. Are they optimized for your experiments here?
2. The theoretical analysis of SimSiam seems unfinished and vague. Some dimensions collapsed, so the rest have equivalent \lambda. How is it able to prove that the rest is optimizing |Lambda - I|?
3. Other than the generalization bound, there is no empirical insight on how this theory can guide better model selection. 
clarity,_quality,_novelty_and_reproducibility: The theorems are not well formalized. It is recommended to have all definition right after the theorems.
No experimental details.
summary_of_the_review: This paper theoretically studied the connection between inter-class distance and the generalization bound for non-contrastive SSL methods. The theory is not solid and has limited insight.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper2523/Reviewer_W5n5
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The paper focuses on the generalization of the non-contrastive self-supervised learning method (Barlow Twins and SimSiam). The authors argue that the generalization is affected by the output dimension and the number of latent classes. They support their findings through experiments on the CIFAR dataset.

strength_and_weaknesses: Strength:
1. As far as I know, this is the first paper analyzing the generalization for Barlow Twins and SimSiam.
2. The paper has a clear motivation. 

Weakness:

The major weakness is Equation (6), where the authors assume that the encoder $f$ is Lipschitz continuous with respect to augmentation from the same latent class. Or in other words, the assumption assumes that the encoder has a good cluster effect on the input $x$ from the same latent class. This assumption is not sound and may not be true. The author does not have a clear definition of $f$. If I understand correctly, $f$ should be one of the optimal solutions for the pretraining loss. Then, here are two concerns (1) in practice, the images from different augmentation methods may have a large pixel-wise euclidean distance, e.g. different rotations from one image. It is hard to guarantee a small Lipschitz constant here. For example, $\epsilon$ may be larger than ¼, and then, all the bound in the theorem is vacuous. (2) In Equation (6), why the Lipschitz condition works for the whole data distribution? Here is a generalization gap that in practice, we only have finite pre-training data. Then, how to guarantee the Lipschitz condition for unseen data. The author needs to provide a generalization complexity analysis in the pre-training stage as well like [1,2]. In [1,2], they have a thorough study of the encoder property and pre-training generalization, while this paper misses this important part.  
From my perspective, the assumption, that the encoder has a cluster effect, is very strong. All the following results (Theorem 1,2,3) are based on this assumption straight-forwardly. The assumption is the most critical part of this paper, while the authors do not provide enough intuition, analysis, or experiments to support the Lipschitz assumption. 

Here are some other weaknesses:
1. The authors claim that "the generalization is affected by the number of latent classes". However, the intuition and definition of latent class are missing. What is the mathematical definition of the latent class? For example, in CIFAR10, the bird is a class label, while in Imagenet, different kinds of birds have different labels. 
2. There are only small-scale experiments. The authors may consider Imagenet. 
3. There is no empirical support for the analysis of SimSiam (from Equation (14) to Equation (18)). A correlation matrix visualization misses here. 

[1] Arora, Sanjeev, et al. "A theoretical analysis of contrastive unsupervised representation learning." arXiv preprint arXiv:1902.09229 (2019).

[2] HaoChen, Jeff Z., et al. "Provable guarantees for self-supervised deep learning with spectral contrastive loss." Advances in Neural Information Processing Systems 34 (2021): 5000-5011. 

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper does not have clear intuition and definition about encoder $f$ and latent classes.

Quality: I checked the proof in the appendix and it looks good to me.

Novelty: The novelty is marginal from my perspective. 

Reproducibility: This is a theory paper and I believe the experiments part can be reproduced. 

summary_of_the_review: The paper has a powerful clustering assumption, but the authors do not have enough evidence to support it. If we cannot guarantee the assumption, the results will be vacuous. Thus, I think the paper misses some crucial components and is not ready to publish. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
details_of_ethics_concerns: NA
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper2523/Reviewer_r6sV
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper tries to unifies the loss functions of non-contrastive SSL methods (i.e., SimSiam and Barlow Twins) into one formulation $\min \|\Lambda-I_d\|$, where $\Lambda$ is the correlation matrix $\mathbb{E}[f(\hat x_1)f^\top(\hat x_2)]$, $f(\hat x_1)$ and $f(\hat x_2)$ are outputs of the two towers and $x_1$ and $x_2$ are sampled from *different* latent classes of the data. It further shows that $ \|\Lambda-I_d\|$ is also the upper bound of generalization loss of the downstream tasks (i.e., classification of the latent classes), and thus establishes the connections. Initial experiments in CIFAR show that the derived bound $\mathcal{L}_{bound}$ involving $\|\Lambda-I_d\|$ indeed correlates with downstream tasks (i.e., KNN accuracy) empirically. 
strength_and_weaknesses: Strength 
1. Finding a unified loss function for non-contrastive SSL to explain its feature learning mechanism is an important question to address. 
2. Some theoretical finding is interesting (e.g., relates $\|\Lambda-I_d\|$ with generalization error), and verified in small-scale dataset. 

Weakness
1. Questionable theoretical assumptions.
2. Strong assumptions. 
3. Theoretical argument is loose / imprecise. 


clarity,_quality,_novelty_and_reproducibility: 1. Potentially inconsistent theoretical assumptions.

One of the major problem is that some theoretical assumptions are inconsistent. Note that there are two different measurement of data augmentation, one is $\epsilon$ in the non-overlapping condition of views / augmentations of different latent classes (Eqn. 7), while the other is $\sigma$ in Eqn. 16-17, which is not defined in this paper, but presumably comes from Tian et al. (2021). The question is, how do $\sigma$ and $\epsilon$ related to each other? 

The deeper question is that, since the paper leverages the theoretical analysis of Tian et al. (2021), the paper should also assume all of its three assumptions, including isotropic data and augmentation (see Assumption 2 in Tian et al (2021)). The question is, how does these assumptions interact with the non-overlapping assumption (Eqn. 7)? Are they compatible with each other? If there exists some cases so that they are compatible, will these cases become too specific to be useful? I would argue the paper not to be accepted if the assumptions cannot be made consistent and discussed thoroughly.  

2. Strong assumptions. 

The non-overlapping data augmentation assumption for different latent classes (Eqn. 7) may not be satisfied in real-world scenarios, in which large augmentations are often used. Large augmentation could make views of two distinct samples very close in the input space (e.g., two different outdoor categories get cropped out except for sky). Is that possible to derive the bound with a weaker version of this assumption (e.g., views of different latent classes overlap with low probability, as in population augmentation graph https://arxiv.org/pdf/2106.04156.pdf)? 

3. Theoretical argument is loose / imprecise. 

In Sec. 4.2, the argument of SimSiam belonging to the unified framework is loose. E.g., how does authors connect "some dimensions collapse and others tend to have equivalent $\lambda^i$" to "SimSiam optimizes $\|\Lambda - I_d\|$"? What do it mean to say "in those dimensions that do not collapse, the model optimize $\|\Lambda - I_d\|$"? Considering that the paper claims a unification of different non-contrastive SSL as the key contribution, Sec. 4.2 are critical and imprecise / hand-wavy arguments would simply lead to rejection. I would advise the authors to put Sec. 4.2 into a self-consistent and make a rigorous theorem. 

Other issues: Many symbols are not properly defined. E.g., $\eta$ and $\sigma$ in Eqn. 16-17. Also in Eqn. 1, does $(f(\hat x_1), f(\hat x_2))$ mean inner product? When we write $\|A\|$, what kind of matrix norm do the authors mean? 
summary_of_the_review: Given the comments, the paper still requires substantial works and I would vote for rejection.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough