Semantically-Prompted Language Models Improve Visual Descriptions (2024)

Abstract: Language-vision models like CLIP have made significant strides in vision tasks, such as zero-shot image classification (ZSIC). However, generating specific and expressive visual descriptions remains challenging; descriptions produced by current methods are often ambiguous and lacking in granularity. To tackle these issues, we propose V-GLOSS: Visual Glosses, a novel method built upon two key ideas. The first is Semantic Prompting, which conditions a language model on structured semantic knowledge. The second is a new contrastive algorithm that elicits fine-grained distinctions between similar concepts. With both ideas, we demonstrate that V-GLOSS improves visual descriptions and achieves strong results in the zero-shot setting on general and fine-grained image-classification datasets, including ImageNet, STL-10, FGVC Aircraft, and Flowers 102. Moreover, these descriptive capabilities contribute to enhancing image-generation performance. Finally, we introduce a quality-tested silver dataset with descriptions generated with V-GLOSS for all ImageNet classes.

## introduction language-vision models @xcite have made significant progress in zeroshot vision tasks. however, in agreement with @xcite , we hypothesize that their accuracy is limited by a lack of visual concept descriptions that are both expressive and specific, that is, glosses that detail the unique visual characteristics of a concept. in this work, we investigate this hypothesis by creating and testing a new method for producing visual descriptions with pre-trained language models and semantic knowledge bases. high-quality visual descriptions are crucial in tasks such as zero-shot image classification and text-based image retrieval. improved descriptions facilitate the creation of more useful representations. these are essential in producing robust and class / concept wordnet gloss v-gloss (ours) corkscrew a bottle opener that pulls corks. a tool with a spiral blade that is used to remove corks from bottles. brambling eurasian finch. a small brown bird with a black head and a white patch on its chest. broccoli branched green undeveloped flower heads. a green vegetable with a thick stalk and florets that grow in a dense head. to address these challenges, we introduce v-gloss, a novel method that leverages language models (lms) and semantic knowledge bases (skbs) to generate improved visual descriptions -visual glosses. our results demonstrate the effectiveness of v-gloss in improving the performance of zsic systems. we achieve strong improvements compared to prior work on benchmark datasets such as imagenet @xcite (+1.8%), fgvc aircraft @xcite (+2.6%), and flowers 102 @xcite (+1.6%) in the zero-shot setting. additionally, we introduce v-gloss silver, a silver dataset constructed by v-gloss, which consists of a visual description for each imagenet class. we show that v-gloss silver is useful for zero-shot language-vision tasks such as zsic and zscig, comparing favorably to wordnet glosses. ## tasks our main task is to generate a description for a given class or concept. for example, if an image classification dataset has the class dog, we aim to produce a description such as "a dog is a furry, four-legged canine..." we consider such a description to be a specific kind of gloss. we use two downstream tasks to compare methods of generating class descriptions: zero-shot image classification (zsic), and zero-shot classconditional image generation (zscig). in zsic, the goal is to classify an image based on a set of classes, without having seen any labeled images belonging to those classes. the set of classes depends on the dataset. for example, given an image depicting a dog, we aim to predict the class dog. in zscig, the goal is to generate an image that corresponds to a specific class, again without having seen any labeled examples. for example, given a class dog, we aim to generate an image of a dog. in short, zsic is the task of classifying a given image, while zscig is the task of generating an image given a class. both involve classes and images. visual descriptions of classes provide useful information which can facilitate both tasks by making it easier to either recognize or generate images of each class. therefore, we aim to improve performance on both zsic and zscig by developing a novel method to improve the generation of such descriptions. ## related work language models the advent of transformerbased language models has revolutionized many natural language processing tasks @xcite @xcite @xcite . as these models are scaled up by their number of parameters and quantity of training data, they exhibit emergent abilities such as fewshot and zero-shot learning @xcite . language-vision models significant strides have been made in the field of language-vision models such as clip @xcite and align @xcite . these models apply contrastive pre-training approaches on large image-text datasets, leading to improved representation learning for both text and images and enhanced performance on several multi-modal tasks @xcite . further advancements have been achieved by scaling up pre-training and incorporating auxiliary training objectives @xcite . ## method we begin by describing how we map classes to concepts in a semantic knowledge base (skb), to leverage the concept-specific information the skb contains. we then introduce our novel method v-gloss, which has two variants, normal and contrastive. we conclude by describing the construction of v-gloss silver, a set of class descriptions produced using v-gloss. ## evaluation in this section, we present our evaluation of v-gloss, alongside comparable methods. we describe our datasets, evaluation metrics, baselines, previous methods, and experiments. to ensure robustness, we report the mean over five random seeds in tables 4 and 5 . class false positives contrastives african elephant tusker (44), asian elephant (6) tusker, asian elephant notebook laptop (22), desktop (10), space bar (2) laptop, desktop, space bar ## discussion when looking at our results, a pertinent question arises: why does an skb, such as wordnet, help us do better on tasks related to vision? in this section, we formulate two insights on how the synergy between skbs and lms supports our improvements. insight #1: skbs represent concepts precisely when lms are prompted with higher-quality context, they produce better output @xcite . wordnet provides a precise representation of a class and its relationship to other classes, leaving minimal room for ambiguity. afterward, we can prompt an lm with this precise information to produce unambiguous and high-quality class descriptions. insight #2: semantic similarity is a useful proxy for visual similarity wordnet models lexical semantics as a tree (see figure 3 ), with synsets as nodes and is-a relationships as directed edges. the distance between different nodes reflects the level of semantic similarity, and is by extension an indicator of the level of visual similarity between synsets. alligator and crocodile are semantically similar because they are both kinds of crocodilian, but they are visually similar as well (see ## conclusion this study concentrates on generating visual class descriptions for zero-shot vision tasks. we employ a novel method that combines pre-trained language models (lms) and semantic knowledge bases (skbs) to create high-quality visual descriptions. our findings suggest that the semantic information from an skb can condition an lm to generate improved visual descriptions which yield higher accuracy and expressiveness. we also show that our contrastive algorithm improves fine-grained discrimination between similar concepts. the integration of skbs with lms reveals partially latent knowledge about visual attributes in the latter and demonstrates a significant interplay between the linguistic and visual domains. these results also pave the way for future exploration into leveraging text-only lms in multi-modal tasks.