Do Vision-Language Models Understand Compound Nouns? (2024)

Abstract: Open-vocabulary vision-language models (VLMs) like CLIP, trained using contrastive loss, have emerged as a promising new paradigm for text-to-image retrieval. However, do VLMs understand compound nouns (CNs) (e.g., *lab coat*) as well as they understand nouns (e.g., *lab*)? We curate Compun, a novel benchmark with 400 unique and commonly used CNs, to evaluate the effectiveness of VLMs in interpreting CNs. The Compun benchmark challenges a VLM for text-to-image retrieval where, given a text prompt with a CN, the task is to select the correct image that shows the CN among a pair of distractor images that show the constituent nouns that make up the CN. Next, we perform an in-depth analysis to highlight CLIPs’ limited understanding of certain types of CNs. Finally, we present an alternative framework that moves beyond hand-written templates for text prompts widely used by CLIP-like models. We employ a Large Language Model to generate multiple diverse captions that include the CN as an object in the scene described by the caption. Our proposed method improves CN understanding of CLIP by 8.25% on Compun. Code and benchmark are available.

## introduction a compound noun (cn) is a noun formed from two or more words combined to create a single noun with a new meaning. a cn usually combines two nouns (noun + noun, e.g., paper towel) or an adjective and a noun (adjective + noun, e.g., full moon); however, there exist more types, and we show an exhaustive list with examples in appendix a.3. for the scope of this paper, we focus primarily on the noun + noun type. interpreting the meaning of cns by decoding the implicit semantic relation between their constituent nouns has attracted great interest in natural language processing (nlp) for decades @xcite . this task requires systems to move beyond memorization as cns are continually emerging, with new combinations frequently appearing @xcite . pre-trained language models (plms) that are trained on vast amounts of text and acquire broad semantic knowledge in the process have shown impressive performance in interpreting cns, including unseen cns @xcite . the improvements can also be partly attributed to the transformer architecture, which by design computes a word representation as a function of the representation of its context @xcite . though extensively studied in nlp, whether modern vision-language models (vlms) understand cns is under-explored. open-vocabulary vlms foot_1 like clip @xcite , trained using a contrastive loss between image-caption pairs, have become the go-to choice for imageto-text (zero-shot classification) and text-to-image retrieval @xcite . however, recent work shows that clip-like vlm models often act as bag of words and lack understanding of relationships between objects and attributes @xcite . for example, prior works explore the failure of vlms to understand spatial relationships between two objects in the image through the caption (e.g., "left of") @xcite or the binding of a verb with its corresponding object (e.g., "running tiger"). to the best of our knowledge, evaluating a vlm's understanding of the semantic relationship between nouns to interpret cns hasn't been explored in literature. ## retrieval with example captions fig. 1 illustrates our proposed approach. as discussed earlier, the standard approach for text-toimage retrieval using class names is to hand-write several prompt templates (e.g., "a photo of a class name."). we propose an alternative frameworkretrieve with example captions. our framework is zero-shot and requires no further training. given a compound noun c, we ask an llm to generate 5 diverse captions, where each caption has the compound noun c as an object in it. the generated captions should have c in diverse settings with diverse adjectives and verbs. we instruct gpt-4 (openai, 2023) with the following prompt to generate the captions: return a list of 5 diverse captions with a com-pound_noun in a photo. the captions should be a maximum of 10 words and one-liners. all 5 captions should describe the compound noun in diverse settings with different verbs and actions being performed with the compound noun. an example output for "chicken we first generate 5 diverse captions describing 5 diverse scenes, with the compound noun as an object in it. these captions are then used to build 5 custom text prompts for text-to-image retrieval, and the image with the highest mean similarity to all 5 prompts is then selected for retrieval. we then build a prompt for our vlm separately with each of the captions as follows to get 5 final prompts: "a photo of a {class name}. an example of {compound name} in an image is {caption}". next, we calculate the mean similarity of an image @xmath0: our core hypothesis builds on existing work in using language as an internal representation for visual recognition, which creates an interpretable bottleneck for computer vision tasks @xcite . instead of querying a vlm with just the compound noun, employing language enables us to compare to any words flexibly. since interpreting compound nouns is easier when provided with proper context in example sentences, getting exposed to diverse keywords through examples makes the image with the compound noun a strongly activating image while the distractors are lowly activating. taking an example from fig. 1 , keywords like player and wooden obtained through diverse captions make the original image more activating than its distractors. our proposed method also improves performance on benchmark text-to-image retrieval datasets, and we provide additional results in appendix a.2. ## experiments and results baselines. for our baselines, we employ the original clip @xcite , openclip @xcite , align @xcite , al-bef @xcite , blip @xcite and metaclip @xcite . all these methods are trained with contrastive learning on image-text pairs. we also employ clip w/ desc @xcite , which adds image descriptors to the prompt for retrieval. finally, we also ablate with clip rev. where we switch the order of nouns in the compound noun. we prompt gpt-4 with a temperature of 0.1 and top-p of 1. results. ## results analysis to perform an in-depth analysis of the results, we first perform an mturk study to divide all cns in compun into 3 main categories as illustrated in fig. 2 : 1.cns that clearly show either of their constituent nouns in the picture. in this case, one noun acts as an attribute to the other, changing its visual minimally, but is not visible itself (e.g., coffee grain). 2.cns that clearly show both the constituent nouns in the picture. this is the same as 1., but both nouns are visible (e.g., snow ball) and 3. cns that show none of the constituent nouns in the picture and make up a completely new cn (e.g., cricket bat). the 3 settings have 199, 106, and 95 instances in compun. fig. 3 compares the number of incorrectly predicted instances in compun across these 3 categories. clip makes the highest number of mistakes in the first category, which also indicates clips' limited understanding of such cns, which can also be interpreted as attributed cns. such cns are also emerging in nature @xcite , and correctly interpreting them is a long-standing problem in nlp. on the other hand, clip makes the least mistakes in the third type, indicating that clip might have acquired adequate knowledge about unique objects in its pre-training stage. fig. 4 shows that results on the compun benchmark are superficially low -similarity scores for correct predictions are ≈25 +-2% (in contrast to imagenet retrieval with ≈82%). we perform retrieval, treating the entire benchmark images as negatives, and achieve a score of 12%, a 66.25% drop. ## conclusion this paper presents the first study of vlms in interpreting cns. we curate compun, a novel benchmark with 400 unique cns, and show that clip has a limited understanding of cns where one of the two constituent nouns acts as an attribute to the other. next, we present a novel approach that moves beyond generic template-based prompts and leverages an llm to generate diverse captions describing scenes with the cn as an object in the scene. our proposed method improves the performance of clip on compun significantly.