Delving into the Openness of CLIP (2023)

Abstract: Contrastive Language-Image Pre-training (CLIP) formulates image classification as an image-to-text matching task, i.e., matching images to the corresponding natural language descriptions instead of discrete category IDs. This allows for open-vocabulary visual recognition, where the model can recognize images from an open class set (also known as an open vocabulary) in a zero-shot manner. However, evaluating the openness of CLIP-like models is challenging, as the models are open to arbitrary vocabulary in theory, but their accuracy varies in practice. To address this, we resort to an incremental perspective to assess the openness through vocabulary expansions, and define extensibility to measure a model’s ability to handle novel classes. Our evaluation shows that CLIP-like models are not truly open, and their performance deteriorates as the vocabulary expands. We further dissect the feature space of CLIP from the perspectives of representation alignment and uniformity. Our investigation reveals that the overestimation of openness is due to confusion among competing text features, rather than a failure to capture the similarity between image features and text features of novel classes. We hope that our investigation and analysis will facilitate future research on the CLIP openness issue.

## introduction an intrinsically open mechanism for visual recognition @xcite has always been a shared goal in the computer vision community @xcite @xcite . this mechanism requires models to maintain flexibility to cope with the scaling of the recognition target, where both input images and the corresponding classes will dynamically expand according to actual needs. for example, in medical diagnosis @xcite , new diseases emerge constantly, and in e-commerce, new categories of products appear daily @xcite , which cannot be predefined in a finite, fixed class set. faced with the challenging task of open-world recognition, contrastive language-image pretraining (clip) @xcite and its openvocabulary learning paradigm demonstrate superiority over traditional supervised classifiers @xcite . clip pretrains a vision-language model on web-scale collections of image-text pairs, learning semantic alignment between images and corresponding textual descriptions. during inference, it formulates image classification as an image-to-text matching task, where the set of class names serves as a vocabulary, and textual prompts like "a photo of a [classname]" are curated as class descriptions for images. by varying the [classname] placeholder and computing the similarity between class descriptions and images, clip can identity the most suitable class name and predict it as the target class. this approach allows clip to operate with arbitrary vocabularies and adapt to novel classes by expanding the vocabulary, enabling zero-shot inference for new input images and classes. nevertheless, previous evaluation protocols for clip models only assess their accuracy on static, closed vocabularies from downstream datasets, leaving their actual performance on open tasks in the shadows @xcite . in this work, we delve into openness, the intriguing yet underexplored property in clip-like models @xcite @xcite , and present a novel protocol for evaluating the openness from an incremental view. specifically, we define a metric of extensibility to measure a model's ability to handle new visual concepts through vocabulary expansion. different from previous metrics, our metric explicitly models the dynamics of the real open world, and formulates the empirical risk of clip when new vocabularies incrementally emerge. additionally, we define a metric of stability to explore how stable the model's predictions are for old classes when new classes are introduced, which provides a tool to analyze the compatibility between different classes. using our protocol, we conduct a systematic and comprehensive evaluation of clip-like models. our experimental results based on extensibility show that clip and its variants have a significant drop in accuracy as the vocabulary size increases. for example, clip (rn101) on cifar100 experiences a 12.9% drop in accuracy when the vocabulary size expands from 5 to 100. this indicates that the limited zero-shot capability of clip-like models is inadequate for supporting their deployment in the open world. what's worse, through an analysis of the prediction shift during vocabulary expansion, we find that the performance of clip can be dramatically reduced by adding only three adversarial class names into the vocabulary, exposing the model's poor stability and security risks. furthermore, we investigate the representation space of clip-like models via three metrics: margin, inter-modal alignment, and intra-modal uniformity. our results show that the small margin between positive and negative class descriptions leads to prediction shifting when competing class features appear. therefore, enforcing the distinguishability of class features increases the margin and improves the stability of these models. in summary, our contribution is threefold: first, to the best of our knowledge, we are the first to systematically quantify the openness of clip, for which we design the evaluation protocol and two indicators of extensibility and stability. second, we conduct extensive experiments on clip-like models based on our protocol and find that their openness is overestimated and their performance declines as the vocabulary expands. finally, we analyze the feature space of clip from the perspectives of representation alignment and uniformity, observing that the uniformity of the textual space is critical for better extensibility. ## related work contrastive language-image pre-training and open-vocabulary learning. clip @xcite introduces the paradigm of open-vocabulary learning and learns transferable visual models from natural language supervision. the clip model con-sists of an image encoder and a text encoder, which are utilized to encode image-text pairs into a joint feature space for learning the semantic alignment of vision and language. the paired images and texts are pulled together in the feature space, while the others with dissimilar semantics are pushed apart via a contrastive loss. after pre-training on large-scale image-text pairs, clip is able to map images to their corresponding language descriptions, which makes visual recognition generalize in the wild. recent studies further improve clip by using more pre-training data @xcite , incorporating self-supervision @xcite , fine-grained supervision @xcite , and widespread supervision @xcite to pretraining. another line of recent studies @xcite @xcite adopts seq2seq generation instead of contrastive discrimination framework to achieve open-vocabulary recognition. we leave the investigation of their extensibility for future work. ## openness, extensibility, and stability in this section, we first review clip's visual recognition paradigm and demonstrate how it realizes open-vocabulary image classification through vocabulary expansion ( § 3.1). to quantify the actual performance of clip-like models as the vocabulary expands, we define the metric of extensibility and propose a systematical evaluation protocol ( § 3.2). ## conclusion in this paper, we evaluate the extensibility of cliplike models for open-vocabulary visual recognition. our comprehensive study reveals that as the vocabulary expands, the performance of these models deteriorates significantly due to indistinguishable text features among competing classes. we hope that our investigation and analysis will facilitate future research on the clip openness issue.