Text-Guided Alternative Image Clustering (2024)

Abstract: Traditional image clustering techniques only find a single grouping within visual data. In particular, they do not provide a possibility to explicitly define multiple types of clustering. This work explores the potential of large vision-language models to facilitate alternative image clustering. We propose Text-Guided Alternative Image Consensus Clustering (TGAICC), a novel approach that leverages user-specified interests via prompts to guide the discovery of diverse clusterings. To achieve this, it generates a clustering for each prompt, groups them using hierarchical clustering, and then aggregates them using consensus clustering. TGAICC outperforms image- and text-based baselines on four alternative image clustering benchmark datasets. Furthermore, using count-based word statistics, we are able to obtain text-based explanations of the alternative clusterings. In conclusion, our research illustrates how contemporary large vision-language models can transform explanatory data analysis, enabling the generation of insightful, customizable, and diverse image clusterings.

## introduction exploratory data analysis (eda) is crucial in the comprehension and analysis of data @xcite . clustering arises as a cornerstone eda methodology, facilitating the grouping of similar data objects into coherent groups. a dataset of images, for example, can be clustered based on semantic similarities between the shown objects. nevertheless, within applied contexts, variations in user requirements or foci demand distinct clustering formations. one might, for instance, cluster a dataset of cards by rank or by suit (see figure 1 ). in such circumstances, it is advantageous to derive multifaceted insights into a dataset from diverse perspectives. current approaches in alternative image clustering either rely on image-based features @xcite or utilize text through image-text bi-encoders, often with architectures resembling clip @xcite . these methods, while powerful, neglect the rich insights that can be extracted by models explicitly trained to retrieve specific aspects of information from images using text (e.g., visual question answering (vqa) models @xcite ). @xcite demonstrate the effectiveness of using generated text descriptions to improve standard image clustering tasks, i.e. in scenarios where a single clustering structure is expected. we aim to use image-to-text models to obtain alternative clusterings. by encoding visual content into text, similarity dimensions beyond the visual features can be explored, potentially revealing interpretable relationships. we introduce tgaicc (text-guided alternative image consensus clustering), a clustering method that uses the output of multiple image-to-text models to obtain alternative clusterings. tgaicc incorporates vqa models to generate multiple textual descriptions of images and then clusters the images based on the generated natural language descriptions. we identify similar clusterings using their mutual information, group them using hierarchical clustering and aggregate them using consensus clustering to form refined, alternative clusterings. our experimental setup employs four widely used alternative image clustering datasets, each possessing two or three ground truth labelings (e.g., playing cards clustered by rank or suit). we compare tgaicc against baselines for alternative clustering using image-only features and baselines that make use of the generated text. our experiments demonstrate the following key findings: methods clustering the generated text outperform methods based on image features on these alternative clustering datasets, underscoring the power of textual representations in capturing diverse aspects of similarity. further, tgaicc, on average, achieves superior results across the evaluated datasets and metrics when compared to all other methods, highlighting the effectiveness of our framework in leveraging image-to-text models to uncover alternative and insightful clusterings. lastly, we can better interpret the clusterings by explaining the content using word statistics. our case study on the cards dataset shows that text provides an opportunity to obtain an informative overview of the data. in summary, our research provides the following contributions: ## related work this work builds upon image clustering, consensus clustering, and alternative clustering approaches. we provide a brief overview of these relevant areas and describe the necessary background. ## tgaicc we use image-to-text models, specifically models that are able to describe specific aspects of information from images in order to obtain different clusterings. thus, we design prompts to perform vqa. it is well known @xcite that responses to seemingly semantically equal prompts might vary heavily. thus, we use multiple formulations of each prompt and aggregate their clusterings afterward. figure 2 gives an overview of the process. ## experiments in this section, we introduce the experimental setup. afterwards, we discuss the main results obtained, highlighting the performance of tgaicc and textbased methods. additionally, we provide an ablation study, systematically analyzing the impact of various components and prompts on the overall performance. finally, we perform a simple cluster explainability method to get a textual overview of the data. ## conclusion in conclusion, this research introduces tgaicc (text-guided alternative image consensus clustering), a novel approach that leverages prompting to inject domain knowledge and human intuition into the clustering process. the experiments on four common alternative image clustering benchmarks demonstrate that tgaicc outperforms competitive image-and text-based baselines. furthermore, the inherent explainability of text enables a deeper understanding of the underlying data cluster formations. by utilizing textual prompts, we can explicitly guide the clustering process from various angles simultaneously, aligning with human intuition. this approach offers a more comprehensive and flexible way to analyze visual data, revealing insights that might be missed by traditional clustering methods. tary materials to enhance the reader's understanding.