Paper: Image Clustering Conditioned on Text Criteria (ICLR-2024)

Reviewer comment (Experiments):
Since the model size of LLaMA-2 and GPT-4 are too big, the author can give more testing results on smaller models.

Author response:
We conducted an ablation study on object clustering using Llama 2 7B, 13B, 70B, GPT-3.5, and GPT-4 (Figure 3). GPT-4 showed the highest performance, but even the smallest model, Llama-2 7B, demonstrated a comparable performance. Llama-2 7B is one of the smaller large language models among the widely used open-source ones.

Reviewer comment (Experiments):
Some baseline models are missing, such as GCC[1] and TCC[2].

Author response:
Thank you for the pointers. We have added them to the related work section in the revised manuscript.

Reviewer comment (Experiments):
Since the model size of LLaMA-2 and GPT-4 are too big, the author can give more testing results on smaller models.

Author response:
We conducted an ablation study on object clustering using Llama 2 7B, 13B, 70B, GPT-3.5, and GPT-4 (Figure 3). GPT-4 showed the highest performance, but even the smallest model, Llama-2 7B, demonstrated a comparable performance. Llama-2 7B is one of the smaller large language models among the widely used open-source ones.

Reviewer comment (Experiments):
Some baseline models are missing, such as GCC[1] and TCC[2].

Author response:
Thank you for the pointers. We have added them to the related work section in the revised manuscript.

Reviewer comment (Novelty):
The paper is more like an engineering guidance instance of a research work. In other words, the proposed method "violently" clusters the images, all based on pre-trained VLM and LLM. Though I acknowledge the good question and paradigm raised by this work, I feel it is better to incorporate some clustering techniques into the paradigm (maybe in future works).

Author response:
The reviewer makes a great suggestion, namely incorporating existing clustering techniques into the IC|TC paradigm. This is something we had explored, and one approach that somewhat worked was using IC|TC as an initialization scheme for k-means.

Specifically, we view the output of IC|TC as an initial clustering assignment, similar to the smart initial clustering assignment of k-means++. Then, we update the clusters through steps inspired by k-means, as more precisely described by the pseudo-code in the subsequent post. In our experiment with Stanford 40 Action, the clusters stabilized (converged) after 4 iterations. However, we did not observe a significant performance improvement to justify incorporating the result into the paper. Nevertheless, combining existing clustering techniques with IC|TC is indeed an interesting direction of future work that we plan to pursue.

Reviewer comment (Experiments):
The refinement operation also seems a bit violent. It seems that if the user is not satisfied with the current clustering results, the whole process is simply repeated. I wonder if the refinement could integrate the experience learned from the last process.

Author response:
In the refinement process, it is not necessary that all three steps be repeated from scratch. In the Fair clustering experiment of section 4.4, we examined the results of Steps 1 through 3 and determined that it was enough to repeat Step 3 in the refinements.

Reviewer comment (Experiments):
100 samples from 10 location classes might have biases by random sampling. I suggest enlarging the datasets of location and mood so that they would become reliable benchmarks for future studies.

Author response:
The reviewer raises a good point. We have additionally labeled 1000 images for the 'location' and 'mood' clustering tasks for the Stanford 40 action dataset and will release the label data so that it can be used as benchmarks for future studies. Based on these additional labels, we updated the evaluation numbers, and we found them to be similar to the prior numbers that we obtained with fewer labels.

Reviewer comment (Writing):
What is the difference between sending the image description in step 1 and the main object in step 2 into step 3?

Author response:
The reviewer raises a great question, and this is indeed something we investigated ourselves. As an example, let us suppose that the image contains a singing man and an applauding audience and that we are using the text criterion 'action'. We may safely assume that, due to the expressive output from the VLMs, the output of step 1 will contain all information related to "singing" and "applauding." However, the output of step 2a, the raw label, may only capture "singing". Now, suppose that after step 2b, where the cluster names are obtained from the raw labels, there is a cluster name relating to 'applauding' but none directly related to 'singing'. Then, it is necessary for the LLM to be provided with the full textual description, not just the raw label, to properly assign the image to the cluster 'applauding'. In the initial stage of our work, we tried both methods and empirically observed that providing the output of Steps 1 to Step 3 showed superior performance compared to providing the output of Step 2a to Step 3.

Reviewer comment (Novelty):
There is a related recent work Image Clustering with External Guidance (arXiv 2023) that also leverages the text modality to enhance image clustering, which the authors are encouraged to include in the related works.

Author response:
Thank you for this pointer. We have incorporated the discussion of this concurrent work into our revised manuscript.

Reviewer comment (Novelty):
I am not convinced that this problem is not merely a form of retrieval based on user specifications.

Author response:
**Response to "I am not convinced that this problem is not merely a form of retrieval based on user specifications."**

The task of clustering requires (i) finding the clusters and (ii) cluster assignment. While image retrieval techniques can certainly be used for cluster assignment, finding the clusters is, in our view, not a form of retrieval. We discuss this point further in our individual response to Reviewer 32Bv.

Reviewer comment (Experiments):
The approach is inherently not testable at scale unless the authors expend a huge amount of human resources.

Author response:
**Response to "The approach is inherently not testable at scale unless the authors expend a huge amount of human resources."**

Evaluating/testing clustering and most unsupervised learning methods at scale is an inherent challenge to the task, but it is possible to carry out an approximate evaluation by sub-sampling a few data points from each cluster and obtaining human labels for those samples. In this sense, an approximate but fair evaluation IC|TC or any comparable image clustering method at scale does not require an inordinate amount of human resources.

We also clarify that the text-criterion refinement of section 3.4 is an optional step and that it does not require a huge amount of human effort. It simply requires that the user roughly examine the clustering results to determine if they are satisfied; in fact, examining the results and fine-tuning hyperparameters is something a user of any clustering method will do in practice.

Reviewer comment (Experiments):
That is probably why the authors choose relatively small datasets of images that have strong structures.

Author response:
**Response to "That is probably why the authors choose relatively small datasets ... I would be more favorably inclined if the authors could show that the proposed method is scalable to millions of images ...."**

We hope our additional large-scale experiment addresses the reviewer's concern.


$\\phantom{X}$

We thank the reviewer for the constructive feedback. We believe we have addressed the reviewer's main concern about scalability. Since we do show that our proposed method is scalable to a quarter-million images, we kindly ask the reviewer to consider raising the score.

Reviewer comment (Experiments):
I would be more favorably inclined if the authors could show that the proposed method is scalable to millions of images as per the state of the art.

Author response:
**Concerns about scalability**

We clarify that our method *is* scalable. In principle, a simple back-of-the-envelope calculation shows that the algorithmic complexity of IC|TC is essentially linear in the number of images. To show that IC|TC is scalable in practice, we conducted an experiment with a quarter-million images during the rebuttal period. (Scaling to a million images, as suggested by the reviewer, would be possible if we had just a few more days to run the experiments.) Specifically, we used the Places dataset [20] and used 50 subclasses, each with 5,000 images, to create a quarter-million dataset. With the text criterion 'place', we achieve 70.5\% accuracy, which is competitive with the state-of-the-art image classification results. (There is some nuance to this comparison that we discuss in the main writing.) We detail the experimental setting and the results in Appendix C of our revised manuscript. We hope this quarter-million experiment alleviates the reviewer's concerns about scalability.

[20] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recognition using places database. 2014.