From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Accept: poster
metareview:_summary,_strengths_and_weaknesses: The paper proposes to go beyond the binary distinction of samples (pos/neg) and leverage correlated non-positive instances (soft neighbors) for representation learning. The reviewers appreciate the idea, writing and results. They request more analyses, metrics, baselines and ablations, but the findings of this paper merit publication even in the current form.
justification_for_why_not_higher_score: While novel enough, the idea is not entirely new, which is reflected in the reviewers' novelty/significance ratings.
justification_for_why_not_lower_score: No reviewers recommend rejection.
note_from_PC: if the above contains the word "oral" or "spotlight" please see: "oral" presentation means -> notable-top-5% and "spotlight" means -> notable-top-25%. As stated in our emails, we are disassociating presentation type from AC recommendations

From: ICLR.cc/2023/Conference/Paper3569/Reviewer_ZPKB
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper proposes to improve self-supervised contrastive learning by leveraging soft neighbours. The main idea is to maintain a candidate neighborhood set and retrieve K nearest neighbors for each sample as its positive examples. A positiveness score is calculated using an attention module and used as a soft weight of the neighbor. Experimentally, they show the proposed method improves the feature representations for many downstream tasks.
strength_and_weaknesses: Strength 

+Using soft neighors is new in contrastive learning. The idea is well motivated and makes a lot sense. 

+The experimental evaluations are sufficient and shows better performance than SOTA

Weakness

-There is no analysis on the memory complexity of the algorithm because a large pool of neighborhood set should be maintained

-How does the size of the neighborhood set affect the performance? 

-The NNCLR result reported in Table 1 (a) is 74.5, while the original NNCLR paper reports 75.6. What is the reason behind that? 
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is written well and easy to understand. 

Quality: the paper is in a good quality with sufficient evaluations following the standard benchmarks. 

Originality: using soft neighbors is new in contrastive learning. 
summary_of_the_review: This paper proposes to leverage soft neighbors to improve contrastive learning, which is new and important.  The experiment also shows promising results. Therefore, I would recommend an accept.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
empirical_novelty_and_significance: 4: The contributions are significant, and do not exist in prior works.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper

From: ICLR.cc/2023/Conference/Paper3569/Reviewer_v9QH
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper proposed a soft neighbor contrastive learning method (SNCLR) in the self-supervised learning(SSL) area. By pointing out the harshness of the binary positive and negative discrimination of the contrastive loss, the author introduces the soft positive by sampling the soft neighbor concept with the positiveness value. As the result, extensive experiments support the new method to achieve state-of-the-art results in the public benchmarks.
 
strength_and_weaknesses: Strength

The paper is clearly written and well organized. 
Although the major idea is extended from the existing works of NNCLR, it still need to be merited.
The experiments are well conducted and the details are clear.

Weakness
1. The difference between equation 2 and the common contrastive loss is the NN extension in both Positive and Negative terms. In the ablation study, it is better to show the individual benefit of the NN. For example, in three experiments: a NN extension is only applied on the Positive. b NN extension is only applied on the Negative c NN extension is applied on both.
2. Since the performance gain in SSL is tend to be saturated, it's better to provide a standard deviation of the accuracy number for some gap which is less than 1% gain.
3. In section 3.2, "We have empirically tried using different structures of this attention module, including using parametric linear project.." comparing to complex computation, how about did some positive mining, a common method used in Deep Metric Learning [i]. which may be more efficient?

4. Some minor issues in the paper
Section 3.1 
"For other N instance.." should be N-1 instance.
"We can also use only positive samples and substitute y1 with z1.." please define z1 in advance.


[i]Xuan, H., Stylianou, A., & Pless, R. (2020). Improved embeddings with easy positive triplet mining. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 2474-2482).
clarity,_quality,_novelty_and_reproducibility: Clarity. As mentioned above, the paper is well-written and organized. Some of the minor issues can be quickly fixed

Quality. The major idea makes sense. The whole paper is in a nice shape.

Novelty. The proposed method is extended from the existing works of NNCLR.

Reproducibility. The paper includes implementation details in the git repo. 
summary_of_the_review: Overall, the paper is in good shape. Please take a look at the weakness. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: Not applicable
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper3569/Reviewer_EMnc
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: A contrastive learning based self-supervised visual representation learning method is proposed in this paper. Upon observing the positive and negative assignments of training samples, this paper proposes to construct a candidate neighbor set where top candidate neighbors are selected as positive samples. When contribution of these positive samples are measured via cross attentions and defined as positiveness. These samples are denoted as soft neighbors to constitute the contrastive loss computation process. Experiments on several benchmarks and evaluation protocols indicate the feature representations of the proposed SNCLR perform well.
strength_and_weaknesses: + The observation that correlated samples are labeled oppositely largely exists in contrastive learning. This is a fundamental issue and less touched in the literature. Compared to the prior SwAV and NNCLR, the proposed SNCLR adaptively mines correlated samples in the candidate neighbor set. The mining process is based on the cross-attention module which effectively measures the sample similarity during the loss computation phase. Both the samples and their positiveness are delved and formulated into the contrastive learning process.

+ Extensive experiments on standard benchmarks, with linear probe, semi-supervised training, transfer to detection and segmentations, with CNN and ViT backbones. Sota comparisons are performed as well. The overall writing is clear and the visualizations / figures are intuitive to represent the algorithm motivations and effectiveness.

-Compared to clustering and nearest neighbor selection, which are conducted in SwAV and NNCLR, the proposed SNCLR achieves better performance in the experiments. While claiming the combination of both advantages is an indirect way to illustrate the SNCLR effectiveness, more analysis shall be taken on elucidating why SNCLR performs better, especially using cross-attentions. 

-Introducing more terms in Eq. 2 are shown to improve the performance. Any other alternatives to substitute current configurations (e.g., cross-attentions, weighted combination of terms)?

clarity,_quality,_novelty_and_reproducibility: The writing is clear, technical quality is good, the proposed method is novel and the code is provided as well.
summary_of_the_review: Overall, this is a good exploration on mining positive samples to support the contrastive learning computation. Sufficient experiments demonstrate the SNCLR is effective on both CNN and ViT encoders via various training settings and datasets.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 8: accept, good paper