Text-Guided Image Clustering (2024)

Abstract: Image clustering divides a collection of images into meaningful groups, typically interpreted post-hoc via human-given annotations. Those are usually in the form of text, begging the question of using text as an abstraction for image clustering. Current image clustering methods, however, neglect the use of generated textual descriptions. We, therefore, propose Text-Guided Image Clustering , i.e., generating text using image captioning and visual question-answering (VQA) models and subsequently clustering the generated text. Further, we introduce a novel approach to inject task- or domain knowledge for clustering by prompting VQA models. Across eight diverse image clustering datasets, our results show that the obtained text representations often outperform image features. Additionally, we propose a counting-based cluster explainability method. Our evaluations show that the derived keyword-based explanations describe clusters better than the respective cluster accuracy suggests. Overall, this research challenges traditional approaches and paves the way for a paradigm shift in image clustering, using generated text.

## introduction psychologists, neuroscientists, and linguists have long studied the dependence of vision and language in humans @xcite @xcite . although the relationship between these modalities is not fully understood, there is a consistent finding: the brain generates a condensed representation to transmit visual information between brain regions @xcite . a widely discussed type of representation is often referred to as "visual language" or "language of thought" @xcite . studies based on these concepts suggest that language can be a crucial driver of visual understanding. for example, children remember conjunctions of visual features better when accompanied by a textual description @xcite , e.g., "the yellow is left of the black". given this relationship between visual perception and language comprehension, the question arises whether an abstract textual representation benefits image clustering. with the significant growth of visual content created online, image clustering has become essential in, e.g., retrieval systems, image segmentation, or medical applications @xcite @xcite . language offers dense, human-interpretable information, providing multiple benefits when clustering (figure 1 ). emerging multi-modal foundation models and large language models (llms), e.g., blip2 @xcite or gpt-3 @xcite , allow to derive a "visual language" from images. in this paper, we propose text-guided image clustering, i.e., deriving a textual representation from images to perform clustering purely based on their text representation. in figure 2 , we outline three approaches to text-guided image clustering. these approaches are structured by the degree of external knowledge introduced into the clustering process. first, caption-guided clustering uses image captioning models to generate brief descriptions of the image content, requiring no external knowledge. in order to inspect the qualities of image and text representations, we compare vision encoder embeddings with tf-idf @xcite and sentencebert (sbert, @xcite representations of the generated text. our experiments show that on a broad set of eight image clustering datasets, text representations on average outperform the image representations of three state-of-the-art (sota) models. second, keyword-guided clustering injects knowledge about the clustering task by prompting visual questionanswering (vqa) models to generate keywords, using the assumption that only a few keywords of interest are necessary to describe each image sufficiently. interestingly, we observe an average performance increase of 5% for tf-idf-based clusterings. third, prompt-guided clustering introduces domain knowledge in the form of tailored prompts for vqa models. quantitatively, we observe another performance increase and qualitatively show that clusters related to the question are formed better. further, we propose to use the generated text for a straightforward counting-based cluster explainability method, generating a keyword-based description for each cluster. our contributions can be summarized as follows: ## related work we approach image clustering in a novel way by generating more abstract text descriptions using image-to-text models. therefore, we discuss how our approach relates to earlier work in image clustering (section 2.1), text clustering (section 2.2) and give an overview of the enabling technology of image-to-text models in section 2.3. ## methodology we describe the formal setup, the experimental setup, and the chosen datasets. ## text-guided image clustering we explore the potential of generated text for image clustering. first, we use standard image captioning and observe that the text representation outperforms the image representation of several models. second, we guide the text generation using vqa models to generate keywords, which we call keyword-guided clustering, and introduce promptguided clustering, where we use domain-specific prompts to elicit relevant properties. third, we use the generated text for cluster explainability, obtaining keyword-based descriptions for each cluster. ## broader impact we believe there is a lot of unused potential for text as an abstraction in image clustering. text as a proxy for "meaningful" clustering. clustering research aims to find meaningful clusters. in general, it is an open question to define what meaningful exactly stands for, some researchers even call it an ill-posed problem. we argue that text is a good proxy to express meaningfulness as it is based on the natural human form of communication. this is a novel viewpoint on the task of image clustering aligning with research methodologies in the clustering community, where clustering methods are commonly benchmarked with datasets that have human-annotated textual labels as ground truth. our research contributes to the discussion about meaningful clustering by showing cluster acc sem sbert sim. tf-idf sbert truth pred truth pred stl10 87 98 100 100 100 100 imagenet10 94 99 30 30 100 100 cifar10 91 97 90 90 100 100 sports10 99 98 50 50 80 80 inaturalist2021 40 48 0 0 91 45 lsun 75 68 70 80 100 100 har 51 56 20 13 87 87 fer2013 46 46 12 12 38 25 ## conclusion in this work, we introduce text-guided image clustering, using image-captioning and vqa models to automatically generate text, and subsequently cluster only the generated text. after applying multiple captioning models on eight diverse datasets, our experiments show that representations of generated text outperform image representations on many datasets. further, we use text to include taskand domain knowledge by prompting vqa models, resulting in additional improvements in clustering performance. we find that it is possible to shape the clustering favorably according to the information given by a specific prompt. additionally, we use the generated text to obtain a keyword-based description for each cluster and show their usefulness quantitatively and qualitatively. while it is difficult to identify background noise or irrelevant features in the pixel space, text is discrete and interpretable. we show that text-guided image clustering often outperforms clustering purely on image information, and provides interpretability. therefore, our research provides insights into the role of text in determining meaningful clusterings.