Paper: Self-Supervision is Not All You Need: In Defense of Semi-Supervised Learning (ICLR-2024)

Reviewer comment (Novelty):
Some claims in this paper are not proper. For example, the authors claim that “These two lines of research (i.e., SELF-SL and SEMI-SL) have progressed largely independent of each other”. However, to my best knowledge, many semi-supervised learning borrows the idea of contrastive learning, which is an important SELF-SL paradigm, e.g., CoMatch: Semi-supervised Learning with Contrastive Graph Regularization (ICCV 21), CLDA: Contrastive Learning for Semi-Supervised Domain Adaptation (NIPS 21), Class-Aware Contrastive Semi-Supervised Learning (CVPR 22), etc.

Author response:
We agree that some methods have tried combining SemiSL and SelfSL training paradigms, we do discuss some such methods in our related work sections. However, most works in either SemiSL or SelfSL limited learning settings avoid comparative analyses with the other category.

Reviewer comment (Experiments):
I’m not sure whether it is fair to directly compare SELF-SL and SEMI-SL. Firstly, it should be noted that in many practical cases, we can easily acquire the pre-trained large model from web, but the abundant data for training such large model are not released due to privacy issue (except the models pre-trained on ImageNet or other public datasets). When we come to the present problem, what we have are small amount of data-of-interests at hand and the available pre-trained model. At this time, we can only use SELF-SL. Secondly, even we have the training data (maybe unlabeled) for pre-trained model, the distribution of these data is different from that (maybe labeled) in the current task. SELF-SL perfectly fits this setting. However, for SEMI-SL, the labeled data and unlabeled data are usually generated from the same distribution, but SELF-SL will probably fail in this setting. Therefore, I think in many cases, SELF-SL and SEMI-SL are not directly comparable, as they have their own advantages in specific setting and application.

Author response:
We agree that there exist practical settings where SemiSL and SelfSL methods are not equivalent. However, the setting outlined in the paper (i.e. availability of large amounts of unlabelled data and small amount of labels)  is also frequently encountered by practitioners, and of broad general interest.

Reviewer comment (Presentation):
The number of tables in this paper is strange. The authors mentioned that “This matches with the empirical observations in Table 4.1.” However, I cannot find Table 4.1 at all.

Author response:
This was the result of a misplaced latex \label tag, we regret this error and thank the reviewer for bringing it to our attention.

Reviewer comment (Evaluation):
As the authors mention, self-SL is typically used to learn "generalizable representations" and is often applied when there is a very large unlabeled dataset. Further, semi-SL is typically benchmarked against in-domain performance which, despite the names of the tasks considered, is what the paper focuses mostly on. Therefore, I think it's not too surprising that semi-SL outperforms self-SL on the set of tasks the authors focus on.

Author response:
> I think it's not too surprising that semi-SL outperforms self-SL on the set of tasks the authors focus on

We believe it is a somewhat novel finding that presence of class labels for ImageNet during training helps on tasks like Oxford and Paris retrieval which have no class overlap with ImageNet.

Reviewer comment (Experiments):
What I think would be more interesting is to understand (1) if semi-SL is still beneficial when using pre-trained models and (2) when one outperforms the other with respect to the unlabeled dataset's size. I.e. I don't think the set of tasks considered in this paper is the truly interesting ones that are relevant to practitioners.

Author response:
> I don't think the set of tasks considered in this paper is the truly interesting ones that are relevant to practitioners

We would respectfully disagree with this assertion, in the context of representation learning, the transferability of the representation and its robustness to different situations is of significant practical importance.

Reviewer comment (Evaluation):
For experiments with OOD, closed-set acc should be reported in addition to AUROC, which are two different measurement dimensions.

Author response:
> closed-set acc should be reported in addition to AUROC

We report closed set accuracy for all models in Table 1.

Reviewer comment (Evaluation):
The experiments on self-supervised learning and semi-supervised learning seem pretty cut and dry, and I don't doubt that both self-supervised learning and semi-supervised learning can be done well on tasks like in-domain, out-of-distribution data, but what does putting the results of these experiments together tell us about what's going on, and where's the connection?

Author response:
> what does putting the results of these experiments together tell us about what's going on, and where's the connection

We broadly compare Semi-SL and Self-SL methods on the questions of robustness and transferability. Each of our experiments tests one aspect of robustness or transferability.

Reviewer comment (Experiments):
There are too few comparative algorithms in this paper. The author uses two algorithms each to represent semi-supervised learning and self-supervised learning, which is destined to yield less comprehensive conclusions.

Author response:
We choose the methods carefully to ensure comparability of results is due to the the differences in Semi- vs Self- supervised learning and not other extraneous factors. e.g. PAWS is inspired by SwAV. So its not just about 2 sets of methods vs other 2.

Reviewer comment (Novelty):
The contribution seems somewhat limited. Apart from the analysis of experimental results, the paper appears to lack new insights. The scope of the topic is broad, but the actual work done is relatively minimal.

Author response:
We are the first to report the significant robustness advantage enjoyed by Semi-Supervised methods over Self-SL methods. Additionally, we establish that Semi-SL methods are at least equally as transferable as SelfSL methods which is novel as well.