Paper: Zero-shot Clustering of Embeddings with Pretrained and Self-Supervised Learning Encoders (ICLR-2024)

Reviewer comment (Writing):
This is the first instance of the "zero shot clustering" task, yet the task has not been formally defined in the manuscript. A formal definition should for example define that the class sets between the feature extractor's training set and the set being clustered as disjoint; this is important to avoid problems like the unfairness of comparing CLIP embeddings on equal grounds.

Author response:
We agree that the omission of a clear definition of “zero-shot clustering” is unfortunate and we will make sure that this will be included in a future version.

Reviewer comment (Evaluation):
Including comparisons against CLIP on the same grounds is unfair to other feature extractors and detracts from the "zero-shot" nature of the task. Because it is not possible to determine if the classes (or even the samples themselves) were present in the CLIP training set, the results on CLIP should be included as extra information and treated as such. The current discussion is hindered by comparing against CLIP directly as readers are not able to properly judge the quality of the feature extractor's generalization.

Author response:
This is a fair point. We are currently aiming to include results using the OpenCLIP model, for which we know what data the model was trained on, and therefore can better interpret the results and how they fit within the “zero-shot” scenario.

Reviewer comment (Experiments):
The hyper parameter sweep (second stage) considers parameters only for Agglomerative Clustering (AC), with only HDBSCAN also getting any parameter evaluation (albeit for a single parameter choice). K-means could have its distance metric changed as well as have an alternative formulation with automatic K (elbow method or even using silhouette score itself). Affinity Propagation also has hyper parameters that could be candidates for a sweep. The lack of experimentation with the other clustering methods raises concerns regarding the final results as AC ends up being the recommended method.

Author response:
Our preliminary analysis showed that tuning the HDBSCAN metric distance did not have a notable impact on performance, hence we prioritized tuning AC over HDBSCAN. After the submission, for completeness-sake we extended our full hyperparameter sweep over the hyperparameters of Affinity Propagation and HDBSCAN, and confirmed that their final performance was not notably affected by this further optimization.

Reviewer comment (Experiments):
Excluding CLIP, what would the authors recommend between using CNN or Transformer-based architectures? It is important to consider that a ViT-B is not computationally comparable to a ResNet50 (which is lighter by 3~4x in FLOPs and parameters).

Author response:
Based on our findings we would recommend using contrastive-learning based approaches, given that both MoCo and CLIP are contrastive-based methods. We do also find that there isn’t a single specific method that is good for both backbone types. Surprisingly, we also find that the supervised encoders are in general very good for clustering, which was not originally expected. These results indicate that there is no single best method at the moment, and that our understanding of SSL methods is not fully complete.

With regards to CNN vs Transformer, our experimental paradigm was not set up to investigate this. As the reviewer points out, a ViT-B requires notably more compute than ResNet50. Our choice of architectures was driven by which pretrained models were readily available in accompaniment to prominent SSL papers. Our results indicate that ViT-B performs better than ResNet-50 when the new dataset is in-domain or a similar domain to the training data, but there is little difference between them for out-of-domain datasets. However as we only investigated two backbones, we do not have sufficient evidence to make any strong conclusions on this point.

Reviewer comment (Experiments):
Is the testing set truly out-of-distribution? For a lot of the dataset tested, it heavily overlap with ImageNet images or CLIP training images. I think it is ok to say "the work benchmarked a lot of datasets" but it is not accurate to say "the work benchmarked a lot of out-of-distribution datasets"

Author response:
We agree with the reviewer that not all of the datasets are necessarily out-of-distribution, as there is partial overlap between the domains Therefore we will reformulate our analysis and instead consider three kinds of dataset groups:
- In distribution: ImageNet-1k, ImageNet-V2 (new), CIFAR-10, CIFAR-100
- Fine-grained: Flowers102, FGVC aircraft, NABirds, iNaturalist
- OOD: ImageNet-R (new), MNIST, fMNIST, SVHN

Reviewer comment (Novelty):
Is this work truly the **first** investigation into zero-shot clustering of SSL feature?: Since 2020, there has been lots of work focusing on image-clustering based on a pretrained (SSL) image encoder. The papers have shown that (1) zero-shot clustering can have non-trivial performance (similar to the result of this paper) (2) Performance can be further improved with different finetuning methods. I think it is important to acknowledge the previous works and reconsider the novelty of this work.

Author response:
While there have been prior methods that combine SSL-features with classical clustering, such as [[2](https://openaccess.thecvf.com/content/CVPR2022/html/Ronen_DeepDPM_Deep_Clustering_With_an_Unknown_Number_of_Clusters_CVPR_2022_paper.html)], these methods have optimised the latent space for clustering first, resulting in the procedure not being zero-shot. We would greatly appreciate any references to the reviewer’s two claims and will accordingly change our manuscript.

Reviewer comment (Writing):
Some issues with definition and terminology: CLIP is not self-supervised, it is rather supervised/language-supervised/weakly-supervised. MoCo/DINO/VICReg have a lot in common. Last year's ICLR best paper candidate has shown a strong duality between the methods, so it would be more accurate to just say they are contrastive.

Author response:
We agree that CLIP can be categorised as weakly supervised learning. However, as the model is learnt with a contrastive learning loss-function, we also argue that it is fair to categorise it as “cross-domain contrastive learning” and thereby as self-supervised learning.

We would also argue that MoCo, DINO, and VICReg are not all contrastive methods, as their loss function is not contrastive. As per [[3](https://arxiv.org/abs/2304.12210] we define DINO and VICReg as self-distillation and canonical correlation analysis approaches, respectively. While [[4](https://arxiv.org/abs/2206.02574)] (which we assume is the paper the reviewer is referring to) argue that VICReg and SimCLR can be made equivalent, the paper also directly states that other methods such as DINO, MoCo, and SimSiam do not perfectly fit this framework.

Reviewer comment (Presentation):
Writing and Presentation: I think the work can be improved a lot, with the current format.

Author response:
We are sorry to hear that the presentation was not satisfactory. We are more than happy to improve the writing and presentation of any part if the reviewer is willing to point these out to us.

Reviewer comment (Novelty):
The paper is primarily a vast hyperparameter search of {pretrained models, clustering method, clustering hyperparams, eval datasets} and an analysis of the results. While the results are useful to use as a reference to pick the best models + clustering methods for a particular task, I fail to see any impactful research contributions. The analysis of correlation between AMI and Silhouette score is useful, but again not strong enough to warrant acceptance.

Author response:
We thank the reviewer for giving us the opportunity to highlight our finding that raw SSL features are not necessarily more clusterable than those from a supervised network is a novel insight, and worth diving deeper into in future research. This challenges the idea that SSL features should be more general than supervised features. We also want to clarify that we do not recommend any specific combination of encoder and clustering method. We instead recommend using a single clustering method—agglomerative clustering—for all tasks.

Reviewer comment (Evaluation):
The results in Figure 1 and 2, especially Figure 1, have very wide error bars, meaning that the comparisons wouldn't be statistically significant. This is probably because of the use of the rank metric -- it would be better to see an aggregated performance metric which isn't as sensitive (like an average), resulting in some statistically significant results.

Author response:
A rank test has fewer priors than a *t*-test and we don't think there is a reason to strongly assume that the delta in the accuracy between two approaches on different datasets would have a Gaussian distribution. Therefore, we are of the opinion that our current approach is better principled, but we would indeed probably get more statistical power by assuming the distance was comparable across datasets.

Reviewer comment (Writing):
Minor: the paper considers CLIP as a self-supervised method, whereas it is typically considered a weakly supervised method since the texts provide noisy supervision.

Author response:
We agree that CLIP can be categorised as weakly supervised learning. However, as the model is learnt with a contrastive learning loss-function, we also argue that it is fair to categorise it as “cross-domain contrastive learning” and thereby as self-supervised learning.

Reviewer comment (Evaluation):
It would be interesting to see a connection between clustering scores and traditional SSL evaluations like k-NN, linear probes, or finetuned results -- do these move together in unison or are the clustering capabilities of a model's representations relatively unrelated to other evaluations?

Author response:
This would be an interesting analysis that we hope to integrate in a future version of the manuscript.

Reviewer comment (Evaluation):
Why does agglomerative clustering perform well, or what makes certain methods perform better? Why does changing the architecture change the rank ordering?

Author response:
These are all good questions. While we may provide some insights into why e.g. agglomerative clustering performs well by going deeper into a set of predictions, we will in general be hard-pressed to provide any guarantees. However, we would also argue that these exploratory studies are important as they highlight a hole in our general understanding of commonly used methods (in this case SSL pretraining), even if we cannot provide all the answers for why this is.

Reviewer comment (Writing):
The paper addresses the concept of zero-shot clustering, an important subject in machine learning. However, there is a need for a clearer definition of the zero-shot clustering problem. Additionally, the paper should thoroughly review and discuss related topics, such as transfer learning or unsupervised domain adaptation, for a comprehensive understanding.

Author response:
We agree that the omission of a clear definition of “zero-shot clustering” is unfortunate and we will make sure that this will be included in a future version. 

We agree that the mentioned fields are highly relevant and the literature review should be extended to include these fields.

Reviewer comment (Reproducibility):
In Section 3, further elaboration is required on the training process of the supervised model. It would be beneficial to include a table outlining different SSL models and their respective configurations for enhanced clarity.

Author response:
We used the pretrained models available in TorchVision v0.16, [[ResNet50](https://pytorch.org/vision/stable/models/generated/torchvision.models.resnet50.html#torchvision.models.ResNet50_Weights )] [[ViT-B](https://pytorch.org/vision/stable/models/generated/torchvision.models.vit_b_16.html#torchvision.models.ViT_B_16_Weights )]. We will make sure that this is clearly stated in the manuscript.

Thank you for the suggestion. We will add a summary table indicating the training configurations of each model to the supplementary materials.

Reviewer comment (Evaluation):
While the paper highlights the improved performance of clustering methods with pre-trained models, it lacks an exploration of the underlying reasons for this phenomenon.

Author response:
In general SSL-pretraining has been found to perform well for downstream tasks such as classification, segmentation, and detection. This was the motivation for investigating how well the features can be clustered, without fine-tuning any part of the network.

Reviewer comment (Evaluation):
Visual comparisons, such as t-SNE plots, are essential in the experiment to provide a more comprehensive evaluation.

Author response:
Thank you for the suggestion. We will provide visual examples of clusters in a future version of the supplementary materials, in order to give the reader a better understanding of what is happening.

Reviewer comment (Evaluation):
The paper mentioned dimension reduction in the experiment but did not discuss the effect of them.

Author response:
The effect of dimensionality reduction is partially covered in section 3.5, covering the hyperparameter search. However, we are interested in further quantifying the effect of dimensionality reduction in a future version of the manuscript.