If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions (2024)

Abstract: Recent works often assume that Vision-Language Model (VLM) representations are based on visual attributes like shape. However, it is unclear to what extent VLMs prioritize this information to represent concepts. We propose Extract and Explore (EX2), a novel approach to characterize textual features that are important for VLMs. EX2 uses reinforcement learning to align a large language model with VLM preferences and generates descriptions that incorporate features that are important for the VLM. Then, we inspect the descriptions to identify features that contribute to VLM representations. Using EX2, we find that spurious descriptions have a major role in VLM representations despite providing no helpful information, e.g., Click to enlarge photo of CONCEPT. More importantly, among informative descriptions, VLMs rely significantly on non-visual attributes like habitat (e.g., North America) to represent visual concepts. Also, our analysis reveals that different VLMs prioritize different attributes in their representations. Overall, we show that VLMs do not simply match images to scene descriptions and that non-visual or even spurious descriptions significantly influence their representations.

## introduction the ability of contrastive vision-language models (vlms) to match related text and images depends significantly on the content of text descriptions @xcite . heuristically, to better identify the related images, humans seek more detailed information about the physical appearance of concepts. many recent works extend this heuristic to vlms and augment the descriptions (also called prompts) with additional visual information (e.g., shape) to more accurately identify the related images @xcite . however, in practice, even a) extract b) explore mistral clip rl slender cat shape color habitat spurious shape generate preferred descriptions how clip represents cats learn clip preferences examine the features figure 1: extract: we align mistral with vlm preferences and generate descriptions that contain features that are important for the vlm. explore: we examine various aspects of these descriptions to identify features that contribute to vlm representations. unlike previous works that focus on a very specific question, our goal is to explore and characterize the features that contribute to vlm representations with the feedback from the vlm itself. in this paper, we propose extract and explore (ex2), a novel analysis method for understanding how vlms represent concepts. instead of directly probing the vlm, ex2 uses reinforcement learning (rl) to align an llm with vlm preferences. our reward function measures the similarity between the concept description and its images. thus, it encourages the llm to generate descriptions that incorporate features that are important for the vlm. then, we can inspect these descriptions from various perspectives to identify common factors that contribute to vlm representations. in this work, we examine the descriptions to understand how often vlms prioritize additional information to represent concepts and how often this information explains the physical appearance of concepts. we use ex2 to analyze seven different vlms on six fine-grained classification datasets. the aligned llm successfully learns features that are important for the vlm and generates descriptions that achieve better classification accuracy than a set of generic descriptions. we find that spurious descriptions (e.g., click to enlarge photo of concept) have a major role in vlm representations despite providing no helpful information about concepts. more importantly, even when vlms prioritize informative descriptions, they significantly rely on non-visual attributes foot_0 like habitat (e.g., north america) to represent visual concepts. in our fine-grained analysis, we find that different vlms represent similar concepts differently, i.e., prioritize different attributes. even the same vlm prioritizes different attributes for different datasets, suggesting that extensive experiments across different datasets are required to draw conclusions about vlms. ex2 is specifically suited to automatically carry out such analysis on a wide range of existing classification datasets. finally, we show ex2's application for hypothesis generation and suggest new research questions based on our findings. our findings encourage further work on imagetext pre-training to address vlms' reliance on spurious descriptions and the alignment between nonvisual information and images. moreover, ex2 provides the future work with a flexible tool to study the impact of different pre-training methods and datasets from various aspects. we summarize our contributions and findings as following: ## extract and explore given the large number of textual features that could contribute to vlm representations (e.g., concept attributes and linguistic patterns), it is very costly, if not impossible, to curate and probe vlms for an exhaustive set of features. instead, ex2 learns to automatically generate concept descriptions that incorporate textual features that are important for the vlm, i.e., extract vlm's preferred descriptions (fig. 2a ). then, ex2 inspects these descriptions for common patterns to identify features that the vlm uses to represent concepts, i.e., explore vlm's preferred descriptions (fig. 2b ). ## experiments in this section, we use ex2 to analyze different vlms at three levels (fig. 3 ). we first validate that the llm learns features that are important for each vlm, and aligned descriptions improve the classification accuracy. second, we show that spurious descriptions contribute significantly to vlm representations despite providing no helpful information. more importantly, at the third level, we find that vlms rely significantly on non-visual attributes to represent visual concepts. we discuss potential correlations between pre-training details and our findings in appendix a.3. we conduct a more fine-grained analysis that reveals even for the same dataset, different vlms prioritize different attributes to represent the concepts. moreover, even the same vlm prioritizes different attributes for different datasets. finally, we show ex2's application for hypothesis generation and suggest new research questions based on our observations. ## additional analysis chatgpt accuracy to assess the accuracy of chatgpt @xcite inspections, we randomly select 50 descriptions from each of the vlm-dataset combinations (2100 in total) and manually check if they are informative and contain visual attributes. then, we compare the results to chatgpt responses. chatgpt correctly classifies the descriptions most of the time and ## qualitative results write a story or narrative inspired by a photo of a acadian flycatcher. how does a photo of a grape hyacinth look like? how does a photo of a lenten rose look like? ## conclusion in this work, we introduce extract and explore (ex2), a novel approach for characterizing features that contribute to vlm representations. ex2 uses rl to align an llm with vlm preferences and generates descriptions that include features that are important for the vlm. ex2 then examines the descriptions to identify the common features that contribute to vlm representations. using ex2, we show that spurious descriptions contribute significantly to vlm representations despite providing no helpful information. vlms also rely significantly on non-visual attributes to represent visual concepts. moreover, different vlms prioritize different attributes to represent similar concepts. even the same vlm prioritizes different attributes across datasets. our results encourage future work to address the significant impact of non-visual and spurious information on vlm representations. ex2 provides future work with a flexible tool to study various aspects of vlm representations.