D ELVING INTO THE O PENNESS OF CLIP
Anonymous authors
Paper under double-blind review
A BSTRACT
Contrastive Language-Image Pre-training (CLIP) has demonstrated great potential in realizing open-vocabulary visual recognition in a matching style, due to its holistic use of natural language supervision that covers unconstrained real-world visual concepts.
However, it is, in turn, also difficult to evaluate and analyze the openness of CLIP-like models, since they are in theory open to any vocabulary but the actual accuracy varies.
To address the insufficiency of conventional studies on openness, we resort to an incremental perspective and define the extensibility , which essentially approximates the model’s ability to deal with new visual concepts, by evaluating openness through vocabulary expansions.
Our evaluation based on extensibility shows that CLIP-like models are hardly truly open and their performances degrade as the vocabulary expands to different degrees.
Further analysis reveals that the over-estimation of openness is not because CLIP-like models fail to capture the general similarity of image and text features of novel visual concepts, but because of the confusion among competing text features, that is, they are not stable with respect to the vocabulary.
In light of this, we propose to improve the openness of CLIP in feature space by enforcing the distinguishability of text features.
Our method retrieves relevant texts from the pre-training corpus to enhance prompts for inference, which boosts the extensibility and stability of CLIP even without fine-tuning.
I NTRODUCTION
The seek for an intrinsically open mechanism of visual recognition (Deng et al., 2009; He et al., 2016) has always been a shared goal in the computer vision community (Scheirer et al., 2013; Geng et al., 2021; Bendale & Boult, 2015).
It requires models to maintain flexibility to cope with the scaling of the recognition target, where both input images and the corresponding classes will dynamically expand according to actual needs.
For example, in medical diagnosis (Razzak et al., 2017), new diseases emerge constantly and in e-commerce, new categories of products appear daily (Xu et al., 2019), which cannot be predefined in a finite class set and remain fixed during inference.
Faced with this challenging open-world recognition problem, traditional supervised classifiers and algorithms have struggled, as they only learn to discriminate limited classes in a closed set, and cannot adapt to the scaling of target classes.
However, the emergence of Contrastive Language-Image
Pre-training (CLIP) (Radford et al., 2021) and its open-vocabulary learning paradigm has reversed this situation.
CLIP models visual recognition as a task of image-text matching rather than the classic image classification.
It is pre-trained on web-scale collections of image-text pairs, learning unconstrained visual concepts from natural language supervision with contrastive learning.
During inference, it devises a textual prompt like “ a photo of a [CLASS] ", where the class token can be replaced by any potential class name from a vocabulary.
The prompt-formed class description with the highest similarity to the input image is predicted as the target class.
This modeling paradigm makes CLIP operationally suitable for open tasks in the real world.
When input images and the target classes change, CLIP can still conduct zero-shot inference by adaptively adjusting the class names in the vocabulary and then modifying the corresponding class descriptions for matching, sparing re-training the entire model on new data like the traditional classification-based methods.
Nevertheless, contrary to the note “ CLIP has a wide range of capabilities due to its ability to carry out arbitrary image classification tasks ” in (Radford et al., 2021), previous evaluation of CLIP is still limited in the closed set, leaving its actual performance on open tasks in shadow.
In this work, we rethink openness, the intriguing but under-explored property of CLIP, and present a protocol for evaluating the openness of CLIP-like models (Radford et al., 2021; Li et al., 2021b;
Mu et al., 2021; Yao et al., 2021; Zhou et al., 2021) from an incremental view.
Specifically, we define extensibility , which essentially approximates the models’ ability in dealing with new visual concepts through vocabulary expansion.
Our experimental results based on extensibility show that CLIP and its variants have a significant drop in accuracy, e.g., 12 . 9% of CLIP (RN101) on CIFAR100 as the vocabulary size expands from 5 to 100 , indicating that the limited zero-shot capability of CLIP-like models is not sufficient to support its deployment in the open world.
Different from previous opennessrelated work, we focus on analyzing how the new class descriptions introduced with vocabulary expansion affect the stability of classification on the old input images.
Our investigation reveals that the small margin between text features of different classes leads to the prediction shift.
To improve the distinguishability of text features and the semantic alignment between images and their textual description, we propose a non-parametric method named Retrieval-enhanced Prompt Engineering (REPE), which retrieves relevant captions from the pre-training corpus to customize the prompt for each class during zero-shot inference.
To summarize, our contribution is three-fold: (1) To our best knowledge, we are the first to systematically investigate the openness of CLIP, for which we design the evaluation protocol and two indicators of extensibility and stability.
Through an analysis of the prediction shift during vocabulary expansion, we find that the performance of CLIP is greatly reduced by adding a small number of adversarial non-target classes, exposing the huge risk of its deployment in the open world.
(2) We further dissect the feature space of CLIP from the perspectives of representation alignment and uniformity, observing that the uniformity of the textual space is critical for better extensibility.
(3) We propose a simple yet effective method, REPE, to improve the extensibility and stability of CLIP without fine-tuning.
O PENNESS , EXTENSIBILITY , AND STABILITY
In this section, we first review CLIP’s visual recognition paradigm based on image-text matching, and then demonstrate how it realizes open-vocabulary image classification in theory by vocabulary expansion ( § 2.1).
To quantify the actual performance of CLIP-like models as the vocabulary expands, we define the metric of extensibility and propose a systematical evaluation protocol ( § 2.2).
The experimental results and further analysis reveal that, as the vocabulary expands, the predictions of CLIP are unstable and prone to drift to the competing class descriptions that are newly introduced, which limits its extensibility and leaves a huge security risk when deployed in real-world applications ( § 2.3).
2. O PENNESS OF CLIP
In contrast to the classic supervised methods (He et al., 2016; Dosovitskiy et al., 2021), CLIP (Radford et al., 2021) models visual recognition as an image-text matching task with self-supervised contrastive pre-training.
Formally, let f be the CLIP model, it takes an image x and a target vocabulary
V ( T ) =
{ w i } of the class names w i as inputs, and outputs the predicted label ˆ y of the image as:
[Equation]
where t i is the textual description of the class name w i in a prompt format, e.g., “ a photo of a w i ”, sim( · , · ) denotes cosine similarity, f T and f
I is the text and image encoder in CLIP, respectively.
Such a modeling paradigm can realize the open-world image classification in theory by extending the target vocabulary V ( T ) to arbitrary degrees.
However, in most previous work (Radford et al., 2021;
Li et al., 2021b; Mu et al., 2021; Yao et al., 2021;
Zhou et al., 2021), CLIP is evaluated with
a fixed
V ( T ) depending on the target classes of the downstream dataset D ( T ) :
[Equation]
where |D ( T ) | denotes the size of the dataset, and I ( · ) is the indicator function.
This vanilla evaluation setting with restricted input images and classes is insufficient for the open recognition tasks, as it
[Figure]
𝑻
[Figure]
𝓥
[Figure]
𝑻
[Figure]
𝓥
[Figure]
𝑵𝑻
[Figure]
𝑵𝑻
[Figure] does not explicitly and systematically models the dynamics of vocabulary expansion during inference, and thus cannot reflect the actual openness of CLIP in the face of real-life class scaling.
2. Q UANTIFYING EXTENSIBILITY OF CLIP FOR THE OPEN WORLD
To quantify the model’s capability in dealing with newly emerged recognition targets, we propose an evaluation protocol and define a metric of extensibility based on vocabulary expansion.
Concretely, we incrementally expand the vocabulary V ( T ) in Eq.
2 by introducing new classes and the associated input images, then evaluate the accuracy after each vocabulary expansion.
These values of accuracy measure the dynamic performance of the model as the openness gradually increases, and their expected average is defined as the model’s extensibility.
In practice, we achieve this expansion by incrementally unioning N disjoint target vocabularies 1 as shown in the upper right panel of Figure 1.
Definition 2.1 (Extensibility) .
Given N disjoint target vocabularies {V ( T ) 1 , · · · , V ( T ) N } , we denote
S N as their full permutation and V ( T ) s i as the i ( th ) vocabulary in a permutation s ∈ S N .
When we union the i ( th ) vocabulary with the previous i − 1 vocabularies, we achieve a vocabulary expansion and obtain V ( T ) s 1 ∪ · · · ∪ V ( T ) s i .
The extensibility refers to the averaged classification accuracy across
N incremental expansions as i increases from 1 to N :
[Equation]
Experimental settings We evaluate the extensibility of CLIP and its variants, including DeCLIP (Li et al., 2021b), SLIP (Mu et al., 2021), Prompt Ensemble (Radford et al., 2021), CoOp (Zhou et al., 2021), on the CIFAR100 (Krizhevsky & Hinton, 2009) and ImageNet (Deng et al., 2009) datasets.
Non-matching methods (Gao et al., 2021; Zhang et al., 2021; Wortsman et al., 2021) like linear probing, etc., are NOT included since they require training a classifier with finite class vectors, and thus are not suitable for class scaling in operation.
To construct the vocabulary, we leverage the underlying superclass-class hierarchical structure of the two datasets (Krizhevsky & Hinton, 2009;
Santurkar et al., 2021), and group the classes belonging to the same superclass into a vocabulary.
There are 20 vocabularies in CIFAR100, each with 5 classes.
For ImageNet, we utilize two superclassclass structures (Santurkar et al., 2021):
Entity13 and Living17.
The former has 13 vocabularies, each [Table] with 20 classes, while the latter has 17 vocabularies, each with 4 classes.
2 Tables in the Appendix A list all the vocabularies in the two datasets.
For each dataset, we calculate Acc-C, the averaged classification accuracy across all single vocabularies, based on Eq. 2:
[Equation]
It represents the original model performance on closed vocabularies.
To calculate the expectation in
Acc-E, we sample 100 × N permutations for N vocabularies and take the average.
Results As shown in Table 1, all models exhibit a clear drop in performance as the openness of tasks increases.
For example, on CIFAR100, compared with the accuracy on closed vocabulary (Acc-C), the accuracy after the vocabulary expansion (Acc-E) of CLIP (RN101) sharply decreased by 12 .
The performance on the data splits in ImageNet is relatively better, with an averaged decline of 2 .
Appendix B provides results of expansion at the dataset level where the expanded vocabularies are from five other datasets.
The performance of CLIP-like models drops even more dramatically by an average of 15 .
3% on generic dataset expansion.
These results demonstrate that the openness of
CLIP-like models is overestimated under the vanilla evaluation mechanism .
Besides, there are some interesting findings: (1) From the perspective of pre-training, introducing a stronger vision backbone (ViT (Dosovitskiy et al., 2021)
v.s. ResNet (He et al., 2016)), widespread supervision (DeCLIP (Li et al., 2021b) v.s. CLIP), and more pre-training data (CLIP v.s. SLIP (Mu et al., 2021)) can improve the extensibility of models on open tasks.
(2) During inference, the performance of CLIP can be boosted by ensembling different prompts (Radford et al., 2021).
(3) CoOp (Zhou et al., 2021) that conducts prompt tuning on all classes of CIFAR100 and ImageNet yields the most extensible results.
However, the prompt tuning method utilizes the predefined category information and training data in the target dataset, which cannot be transferred to real-life open tasks.
2. S TABILITY DURING VOCABULARY EXPANSION
As the vocabulary expansion introduces new classes incrementally, some input images belonging to the previous vocabulary will be incorrectly predicted as new classes, which leads to an accuracy drop and the poor extensibility.
To analyze the prediction stability of CLIP during vocabulary expansion, we introduce the non-target classes .
They do NOT correspond to any input images, only serving as distractors for the target classes.
Based on it, we define conditional classification accuracy as follows:
[Equation]
where V ( NT ) is the non-target vocabulary , i.e., the vocabulary of non-target classes.
The left panel of
Figure 1 gives an illustration of the conditional accuracy.
In Eq.
5, the categories of the input images i n s e c t s a q u a t
i c e l e c t r i c a l d e v
i
c
e s m a mm a l s f
i s h
o u t d oo r t h i n g s t
r ee s f oo d p e
o p l e s m a ll m a mm a l s v e h
i c
l e s f l o w e r
s o u t d oo r s c e n e s v
e h
i c
l e s f r u
i t / v e g e t a b l e
s i n v e r t e b r a t
e s c
[Table]
r n i v
[Table]
r
[Table]
s
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Table] [Table]
[Figure]
[Table]
l
[Table]
v i
[Table]
s h
[Table]
h
[Table]
l d i
[Table]
t s
[Table]
r
[Table]
i
[Table]
t d oo
[Table]
n
[Table]
t d oo
[Table]
s
[Table]
n e s
[Table]
m
[Table]
i
[Table]
r e
[Table] [Table]
i
[Table]
m
[Table]
mm
[Table]
i n
[Table]
t e
[Table]
s p
e
o p l e r e p t
[Table]
s
[Table]
m
[Table]
ll m
[Table]
mm a l s t r ee
s v e h
i c l e sv e h i c
l e s
[Figure]
are limited to the target vocabulary ( ( x , y ) ∈ D ( T ) ), but CLIP is asked to distinguish all categories from a broader vocabulary V ( T ) ∪ V ( NT ) .
In other words, compared to the traditional closed-set classification, CLIP is supposed to reject all the negative categories from V ( NT ) .
The model is required to distinguish visual concepts stably and robustly, rather than making wrong predictions in the presence of other distractors.
Based on Eq.
5, we define the stability of CLIP in the open task:
Definition 2.2 (Stability) .
Given a target vocabulary V ( T ) and M non-target vocabularies {V ( NT ), · · · , V ( NT ) M } , we denote S M as their full permutation and V ( NT ) s i as the i ( th ) vocabulary in a permutation s ∈ S M .
We design the local stability to measure the averaged classification accuracy of CLIP on the given target vocabulary when non-target vocabularies are extended incrementally:
[Equation]
As Eq.
6 only reflects the local stability with respect to a single target vocabulary, we further design the general stability as an average of local stability over a set of target vocabularies to reduce the bias from data distribution and vocabulary sampling.
Specifically, given N vocabularies {V 1 , · · · , V N } , we regard each vocabulary V i as the target vocabulary V ( T ) and the rest V ̸
= i as the non-target vocabularies V ( NT ) , and then formulate the general stability as:
[Equation]
Experimental settings and results The models and datasets adopted for evaluation are consistent with that in § 2.2.
For the calculation of stability, take CIFAR100 with N = 20 vocabularies as an example, we treat each vocabulary as the target vocabulary and the rest are treated as the non-target vocabularies for Eq.
To calculate the expectation in Eq. 6, we sample 100 permutations for M =non-target vocabularies and report the averaged scores.
Table 1 demonstrates the stability of CLIP-like models.
On CIFAR100, the Acc-S of CLIP (RN101) decreased by 13 .
Figure 2a shows Acc-S on CIFAR100 during non-target vocabulary expansion.
Given a closed V ( T ) =
Insects , CLIP (ViT-B/32) achieves an accuracy of 81 . 2% .
However, when the remaining 19 non-target vocabularies are incorporated, the accuracy sharply drops to 57 .
The decrease of Acc-S brought by each introduction of non-target vocabulary indicates that more images from Insects are incorrectly classified into the new vocabulary.
Figure 2b demonstrates the difference between Acc-C and Acc-S for each target vocabulary.
When V ( T ) =
Medium-sized Mammals , CLIP is most easily interfered with by the non-target vocabularies, with a 21 .
08% performance drop.
It suggests that the unstable predictions lead to the poor extensibility of CLIP when new categories are introduced .
Besides, we notice that CLIP performs stably on groups like Flowers , where its Acc-S only declines by 0 .
53% compared to Acc-C.
The different behaviors of different groups indicates that the stability is also influenced by the inherent property of the image categories .
Furthermore, In order to explore the lower bound of the stability, we define the adversarial nontarget vocabulary V ( ANT ) as the non-target vocabulary that reduces Acc-S the most.
Specifically, we maliciously introduce only three new classes and find that the performance of CLIP drops dramatically, e.g., 52.7% accuracy drop on CIFAR10.
It reveals the vulnerability of CLIP and its poor semantic modeling on those objects with higher abstraction levels.
Please refer to Appendix D for details.
D ISSECTING AND IMPROVING THE EXTENSIBILITY OF CLIP
Our experimental results in § 2 expose the unsatisfying performance of CLIP on open tasks.
In this section, we dive into the representation space of CLIP to find the key to understanding and improving its extensibility.
We first point out that the small margin between positive and negative class descriptions leads to the prediction shift when competing text features appear, which thus limits the stability of CLIP ( § 3.1).
Further, we investigate the representation space of CLIP-like models via two metrics of inter-modal alignment and intra-modal uniformity.
The results show that enforcing the distinguishability of text features enlarges the margin and makes models scale more stably ( § 3.2).
In response, we propose a non-parametric method named Retrieval-Enhanced Prompt Engineering (REPE), which boosts the performance of CLIP even without fine-tuning ( § 3.3).
3. S MALL MARGIN LIMITS THE STABILITY OF CLIP
Since CLIP formalizes the visual recognition as an image-text matching task (Eq. 1), each text feature of the class description corresponds to the class vector in traditional classifiers, and the image-text similarity scores are thus analogous to the logits in classification.
Ideally, no matter how the vocabulary expands, for an image, the similarity of the positive pair, i.e., the image with the text specifying the ground-truth class, should be higher than those of the negative pairs, i.e., the image with the texts specifying other classes, to ensure the correct prediction of CLIP on open tasks.
In other words, the margin (Jiang et al., 2019) between positive and the largest negative similarity is a direct contributing factor to the stability.
Unfortunately, the similarity and margin distribution of CLIP does not meet our expectations.
Figureillustrates the averaged cosine similarity of CLIP (ViT-B/32) on 15 classes of CIFAR100.
Similarity over the intact dataset is in
Appendix E.
The diagonal elements represent the similarity of the positive image-text pairs, while the others represent that of the negative ones.
As shown in Figure 4, the similarity histogram of positive and negative pairs has a large overlap.
Its margin in Figure 5 is clustered around zero, leaving the prediction of models at risk of being reversed to the new non-target classes.
For example, as the vocabulary extends from the red box to the green box (diagonal) or the yellow box (horizontal) in Figure 3, more deceptive classes (circle) with negative margin are added, which leads to the prediction shift.
Particularly, the classes belonging to the same vocabulary 3 have higher similarity and smaller margin, which is more likely to be confused with each other.
3. I NTER - MODAL ALIGNMENT AND INTRA - MODAL UNIFORMITY GROUND THE MARGIN
The results in § 3.1 raise a natural question: what vision-and-language feature distribution can maintain a large margin between different classes so that the model can scale stably in the open world?
Here we present two properties of the ideal feature space: First, the text feature of a class name is supposed to stay close to the features of the images it describes, promoting the similarity of positive pairs.
Second, intra-modal features, especially the textual features should be uniformly distributed to preserve maximal information and make the descriptions of competing categories more distinguishable.
Accordingly, we define inter-modal alignment and intra-modal uniformity , two metrics to measure the quality of representations in contrastive learning (Wang & Isola, 2020) for the vision-and-language domain.
Inter-modal alignment calculates the expected distance between
[Table]
flatfish
[Table]
bear
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Figure]
features of positive image-text pairs p pos : ℓ align ≜ E ( x , t ) ∼ p pos ∥ f I
[Equation]
while intra-modal uniformity measures how well the image or text features are uniformly distributed:
[Equation]
where p data-I and p data-T denotes the image and text data distribution, respectively.
Figure 6 and Tablein Appendix F provide quantified loss of alignment and uniformity.
CLIP with only cross-modal contrastive learning results in poor intra-modal uniformity ( ℓ uniform > − 2 . 0 ), especially on the text side.
Introducing intra-modal contrastive learning like SLIP and DeCLIP in pre-training can force both image and text features separate better from classes, reducing ℓ uniform to below − 4 .
As for the prompt tuning method (CoOp), it achieves better ℓ align of 1 . 4 compared to CLIP ( 1 . 5 ) and the lowest ℓ uniform-T of − 3 . 2 .
According to the visualization via Multidimensional Scaling (MDS) (Borg & Groenen, 1997) in Figure 7, the optimization trajectory of prompts is towards the cluster center of the corresponding image features, while dispersing the position of the prompt features, which improves both text uniformity and inter-modal alignment, achieving the best extensibility.
3. M ETHODOLOGY : R ETRIEVAL - ENHANCED PROMPT ENGINEERING (REPE)
In light of the previous investigations, we propose a simple but effective method named Retrievalenhanced Prompt Engineering (REPE) to enforce the distinguishability of text features and the semantic alignment (Cao et al., 2020; Ren et al., 2021).
Recall that the context for each class name is the same in vanilla CLIP-like models (e.g., “ a photo of a [CLASS] ”), making it difficult to discriminate between distinct visual categories because the semantics of each cannot be holistic represented Zhou et al. To remedy this, we propose to customize each class description with
[Figure] [Figure]
refers to image features (-I), while × and ⋆ refers to text features (-T) of CLIP and CoOp, respectively.
of transparent optimization trajec
[Table]
[Figure]
[Table]
diverse captions retrieved from the pre-training corpus as a prompt ensemble.
Specifically, for each class description based on the original prompt, we utilize CLIP to recall the most similar images from the pre-training dataset via image-text similarity then obtain their corresponding captions.
The retrieved captions without class name appearance are filtered out, finally resulting in K captions.
Such a workflow leverages both visual semantics and class name, achieving better performance.
Appendix G shows the instances of the retrieved captions.
They provide the context in which the class name is located and thus have richer semantics.
After that, we encode the retrieved captions and conduct a mean pooling operation among them.
The final text representation is
[Equation]
where rt ij is the j ( th ) retrieved caption for class
i , λ is a weighting factor and is adjusted on the validation set.
The ensemble text representation f REPE T ( t i ) is then adopted as the class anchor for conducting the image classification.
The text representation for the target class is thus shifted towards the representative captions in the pre-training dataset, which alleviates the semantic inconsistency between pre-training and inference.
Experiments We retrieve the images and captions from CC12M (Changpinyo et al., 2021), a subset of the pre-training dataset of CLIP.
The images and captions are pre-encoded within an hour using a single RTX TITAN GPU, then we build their indices for KNN search with FAISS framework (Johnson et al., 2019), which also takes about an hour .
Once the indices are built, we can efficiently search over the dataset according to the query image in less than 5 ms , which is applicable for query-intensive scenarios.
Appendix H provides the detailed process and computational overhead of the retrieval.
Table 2 shows the results of retrieval-enhanced prompt engineering.
The hyper-parameter K isand λ is 0.25.
We find that on all the three datasets, REPE consistently improves the extensibility and stability of CLIP by an average of 1.2% .
We further evaluate the loss of text uniformity and inter-modal alignment for probing the quality of the enhanced representations.
As shown in Figure 6, the former is effectively reduced from − 0 .
8 to − 1 .
and the latter is reduced from 1 . 5 to 1 . 4 , verifying our proposal can improve the class anchor for better extensibility and stability.
Additionally, REPE increases the median value of the margin distribution from 0.005 to 0.01 and pushes the overall distribution towards the positive side compared to vanilla CLIP (Figure 12 in Appendix E).
It indicates that REPE widens the gap between positive and negative class features, making it more difficult to invert predictions with competing classes.
In conclusion, all of these findings support REPE’s efficacy in addressing the openness issue.
It is worth noting that compared to the method that requires computation-intensive pre-training procedure (DeCLIP and SLIP), and the prompt-tuning approach (CoOp) demands access to the downstream target dataset, our REPE is a lightweight framework for the zero-shot inference stage without fine-tuning.
Besides, since REPE is model-agnostic and orthogonal to parameter-tuning methods, it can be also combined with fine-tuning methods like adapter-tuning (Gao et al., 2021), to achieve further performance boost by an average of 0 .
6 on CIFAR100 and ImageNet, which demonstrates the adaptability and superiority of our method.
Please refer to Appendix I for details.
R ELATED WORK
Contrastive language-image pre-training and open-vocabulary learning CLIP (Radford et al., 2021) enables learning transferable visual models from natural language supervision and makes visual recognition generalize in the wild (Zareian et al., 2021;
Gu et al., 2022; Ghiasi et al., 2021).
It is pre-trained on web-scale collections of image-text pairs, learning tremendous visual concepts described by natural language with contrastive learning.
During inference, it devises a prompt like “ a photo of a [CLASS] ", where the class token is a placeholder for any potential class name from a vocabulary, and the class description with the highest similarity to the input image is predicted as the target class.
Another line of recent studies (Li et al., 2021a; Wang et al., 2022; Yu et al., 2022;
Alayrac et al., 2022) adopts seq2seq generation instead of contrastive discrimination framework to achieve open-vocabulary recognition.
We leave the investigation of their extensibility for future work.
Open set and open-world visual recognition Open Set Recognition (OSR) (Scheirer et al., 2013; Geng et al., 2021) requires classifiers to identify images that have not been introduced during training as “unknown”.
The task can be formalized as one-vs-reset classification (Scheirer et al., 2013) or multi-class classification (Jain et al., 2014; Scheirer et al., 2014).
Furthermore, Open World Recognition (OWR) (Bendale & Boult, 2015) raises higher demands that the model must incrementally learn and extend the multi-class classifier as the unknowns are labeled for new class learning.
Contrary to the above research, the CLIP-based Open-vocabulary Recognition (OVR) is unsupervised.
We focus on the model performance on zero-shot inference, without training on the target dataset.
Appendix J provides a more detailed comparison of OSR, OWR, and OVR.
L IMITATIONS AND F UTURE W ORK
To facilitate future research, we analyze the difficulties and possible solutions in this new area.
(1) As we present extensive empirical results and address the weakness of CLIP on vocabulary expansion, its theoretical risk on open tasks is urged to be investigated.
The current evaluation protocol is an approximation of the real open world.
An evolving benchmark could facilitate future research.
For various visual categories, their degree of abstraction, the ease of describing them in natural language, and their density in the data distribution can also influence the extensibility and stability of models, which are worth studying.
(4) Within the CLIP & prompting framework, our REPE can be easily adopted in various downstream visual tasks like open-vocabulary object detection (Zareian et al., 2021), semantic segmentation (Ghiasi et al., 2021), etc.
We leave it for future work.
C ONCLUSION
In this paper, we evaluate the extensibility of CLIP-like models for open-vocabulary visual recognition.
Our extensive investigation shows that the performance deteriorates seriously as the vocabulary expands, which results from the indistinguishable text features among competing classes.
To remedy this, we propose REPE to enhance the textual representations with class-relevant captions retrieved from the pre-training corpus, which boosts the extensibility and stability even without fine-tuning.
A S UPERCLASS - CLASS HIERARCHY FOR VOCABULARY CONSTRUCTION
To construct the vocabulary in § 2, we leverage the underlying superclass-class hierarchical structure of CIFAR100 (Krizhevsky & Hinton, 2009) and ImageNet (Deng et al., 2009), and group the classes belonging to the same superclass into a vocabulary.
Table 3 lists the vocabularies in CIFAR100, which are specified by (Krizhevsky & Hinton, 2009).
There are 20 vocabularies, each with 5 classes.
For ImageNet, we utilize two superclass-class structures, Entity13 and Living17, in (Santurkar et al., 2021).
Table 4 and Table 5 show the vocabularies in ImageNet (Entity13) and ImageNet (Living17), respectively.
The former has 13 vocabularies, each with 20 classes, while the latter has 17 vocabularies, each with 4 classes.
B D ATASET - LEVEL EXTENSIBILITY
The evaluation protocol in § 2 estimates the extensibility and stability within a single task dataset, where the input images and classes during the vocabulary expansion come from the same data distribution.
While the protocol is only an approximation of the real open world, current CLIP-like models have exhibited serious performance degradation.
In this section, we take a step further toward the real open recognition, by conducting vocabulary expansion setting on the dataset level, where the expanded vocabularies are from different datasets.
In this way, the relationship between vocabularies is more uncertain and thus can be viewed as a rigorous stress test for the CLIP-like models.
Specifically, we group all categories in a dataset into one vocabulary.
Afterward, the inputs and classes of the entire new dataset are introduced at each expansion.
Classes in the new vocabulary will be removed if they already exist in the previous vocabularies.
The experiments are conducted with datasets for generic objects, including CIFAR10 (Krizhevsky & Hinton, 2009), CIFAR100 (Krizhevsky & Hinton, 2009), Caltech101 (Fei-Fei et al., 2004),
SUN397 (Xiao et al., 2010) and ImageNet (Deng et al., 2009), and specialized datasets focusing on fine-grained categories, including Flowers102 (Nilsback & Zisserman, 2008), OxfordPets
(Parkhi et al., 2012) and StanfordCars (Krause et al., 2013).
Without loss of the generality, we mergedatasets and evaluate the following dataset compositions:
(1) CIFAR100-Caltech101-SUN (2) CIFAR10-CIFAR100-ImageNet (3) Flowers102-OxfordPets
-StanfordCars
Composition (1) and (2) probe the performance when all the expanded datasets are generic thus the classes in different datasets are semantics correlated, while composition (3) targets scenarios where the coming datasets have little correlation with previous ones.
To eliminate the effect of vocabulary expansion order, we report the average performance of all A 33 = 6 possible trials for each composition.
Table 6 demonstrates the result of the dataset-level expansion.
First , the performance of CLIPlike models on generic dataset expansion drops dramatically.
For example, the accuracy (Acc-E) of CLIP (RN101) decreases by an averaged absolute point of 14 . 2 on the CIFAR100-Caltech101SUN397 composition during expansion, and 14 .
5 on the CIFAR10-CIFAR100-ImageNet composition, respectively.
Due to the existence of subclass-superclass relationship for some classes in different generic datasets, e.g., cat in CIFAR10 and tiger cat in ImageNet, CLIP is extremely unstable on such expansion across generic datasets.
For example, the Acc-S of CLIP (RN101) on the CIFAR10CIFAR100-ImageNet composition is 28.2% lower than Acc-C, indicating the models are prone to be confused about the subclass-superclass relationship.
Meanwhile , the CLIP-like models exhibit much better extensibility and stability on the dataset-level expansion across specialized datasets, e.g., the Flowers102-OxfordPets-StanfordCar composition.
The vocabularies of this composition are intrinsically disjoint in semantics, so the model can be stably extended.
In summary , our investigations on the dataset level expansions along with the task level in the paper show the current
CLIP-like models fail to meet the expectation of conducting real open vocabulary recognition, and we hope our studies can motivate future studies in this direction.
C I NCREMENTAL A CC -E AND A CC -S ON CIFAR
We record the Acc-E (Eq. 3) and Acc-S (Eq. 6) after each vocabulary expansion on CIFAR100 to investigate the effect of increased task openness on CLIP-like models.
Figure 8 shows the Acc-E for 20 trials as new vocabularies are merged incrementally.
The falling lines indicate that the model is either performing poorly on the new input images, or that some images that were correctly identified before are misclassified after introducing the new classes.
[Table]
Figure 9 shows Acc-S of CLIP-like models during non-target vocabulary expansion.
Each sub-figure represents the situation when one vocabulary is selected as the target vocabulary.
As the remaining 19 non-target vocabularies are incorporated and the model is required to recognize the 5 target classes from 100 potential classes, the accuracy drops sharply.
The decrease of Acc-S brought by each introduction of non-target vocabulary indicates that more images from the target vocabulary are incorrectly classified into the new non-target vocabulary by models
[Table]
D A DVERSARIAL NON - TARGET VOCABULARY
In order to explore the lower bound of the stability of CLIP, we define the adversarial non-target vocabulary V ( ANT ) as the non-target vocabulary that reduces Acc-S the most:
[Equation]
To build V ( ANT ) , we refer to the method of adversarial examples generation in the Natural Language
Processing field (Ren et al., 2019) to traverse the words in a large vocabulary, e.g., the vocabulary of nouns in WordNet (Fellbaum, 2000), which are regarded as non-target classes in order to calculate
Acc-S, and then take the most confusing words to form the adversarial non-target vocabulary.
We constrain the size of V ( ANT ) to 3 .
Results in Figure 10 illustrate the performance with nouns in
WordNet and class names in ImageNet as the candidate vocabulary, respectively.
First, we observe a clear performance degradation on both datasets under adversarial attack, e.g., adding bitmap , automobile insurance and equidae leads to an absolute 52 .
7 % accuracy drop on CIFAR10.
Besides, we find that the selected adversarial words are much less concrete than common visual concepts like Flower , indicating the potential reason behind is the poor semantic modeling of CLIP on those objects with higher abstraction levels.
This investigation reveals that CLIP is vulnerable when facing malicious non-target vocabulary , and we hope future work may pay more attention to the robustness of CLIP under open recognition tasks.
[Table]
[Table] [Table]
-S ∆
CLIP (RN101) 65.
-0.1 CLIP (ViT-B/32) 72.
-0.0 CLIP (ViT-B/16) 74.
SLIP (ViT-B/16) 58.
-0.0 DeCLIP (ViT-B/32) 74.
PE (ViT-B/32) 71.
-0.0 PE (ViT-B/16) 75.
[Table] [Table]
E C OSINE SIMILARITY OF IMAGE - TEXT FEATURES ON INTACT CIFAR
Figure 11 illustrates the averaged cosine similarity between image and text features of CLIP (ViTB/32) on all classes of CIFAR100.
The elements on the diagonal represent the similarity of the positive image-text pairs, while the others represent that of the negative ones.
Since every 5 adjacent classes in the figure constitute a vocabulary (superclass), 4 the classes belonging to the same vocabulary have higher similarity and smaller margin, which is more likely to be confused with each other.
Our Retrieval-enhanced Prompt Engineering (REPE) method alleviates this issue by enlarging the margin between positive and the largest negative similarity.
As shown in Figure 12, the median value of REPE’s margin distribution is .
01 (the blue vertical line), which is larger than that of CLIP (ViT-B/32) with .
(the red line).
It indicates that the predictions of REPE are harder to be inverted with competing classes than the original CLIP, thus yielding better performance on the open tasks.
F Q UANTIFIED LOSS OF INTER - MODAL ALIGNMENT AND INTRA - MODAL UNIFORMITY
Table 7 provides quantified loss of alignment and uniformity based on Eq.
8 and Eq.
9 defined in § 3.2.
CLIP with only cross-modal contrastive learning results in poor intra-modal uniformity.
On the vision side, compared with ResNet-101 (He et al., 2016), using a more powerful visual encoder such as ViT (Dosovitskiy et al., 2021) can reduce the loss of image uniformity from − 0 .
57 to − 0 .
Besides, the ℓ uniform of SLIP (Mu et al., 2021) and DeCLIP (Li et al., 2021b) is much lower than
CLIP, indicating their better intra-modal uniformity derived by intra-modal contrastive learning in pre-training, which enforces both image and text features separate better from classes.
As for the prompt tuning (Zhou et al., 2021) method (CoOp), it achieves better ℓ align compared to CLIP and the lowest ℓ uniform-T of − 3 .
G C ASE STUDY OF RETRIEVED CAPTION IN REPE
Table 8 shows some cases of the captions retrieved by our proposed REPE on CIFAR100.
They share the same target of interest with the original prompt, i.e., “ a photo of a [CLASS] ”, but provide the context in which the class name is located and thus have richer semantics.
For example, given a class like bridge , the retrieved captions describe its possible properties (e.g., “golden”, “wooded”), connections to other objects (e.g., “over a mountain river”), etc., yielding more expressive and distinguishable text features of the class.
However, REPE also recalls some low-quality captions.
For example, given the class ray , a large, flat sea fish with a long, narrow tail, in CIFAR100, the caption “Sun Rays Tours: Go Pro captured the rays under water” is retrieved, where the “ray” in the caption 4 See Table 3 in Appendix A.
[Table] [Table]
refers to a narrow beam of light
[Table]
heat
[Table]
etc.
We leave the retrieval with better semantics preservation for future work
[Table]
H C OMPUTATIONAL O VERHEAD OF REPE
The computational overhead of REPE is three-fold (measured a single RTX TITAN GPU): (1)
Computing the text and image embeddings of CC12M dataset (Changpinyo et al., 2021) we used.
It takes about an hour with a single RTX TITAN GPU.
(2) Building indices out of the embeddings for KNN search with the FAISS framework (Johnson et al., 2019).
The cost depends on the hyperparameter setting for the target recall, and the procedure can be finished in an hour for our dataset.
(3) Retrieving the relevant images and corresponding captions for given query images.
Once the indices are built, we can efficiently search over the dataset according to the query image in real time (less than 5 ms for retrieving the top-100 relevant image-caption pairs), which is thus still applicable for query-intensive scenarios.
I I NCORPORATION OF REPE TO DOWNSTREAM FINE - TUNING
Our REPE method is model-agnostic and orthogonal to parameter-tuning methods.
We can combine it with fine-tuning methods like adapter-tuning (Gao et al., 2021) (denoted as CLIP-adapter), to achieve further performance boost.
Concretely, CLIP-adapter adds a tunable adapter network (two linear layers with ReLU) after the last layer of both image and text encoder.
Based on it, we ensemble the original class description (“ a photo of a [CLASS] ”) with the captions retrieved by our
REPE as the final textual input.
The CLIP-Adapter is trained with K-shot training samples and then evaluated on full test splits.
We conduct experiments on CIFAR100 and ImageNet with K=4 and
K=16, respectively.
As shown in Table 9, REPE further improves the performance of adapter-tuning by an average of 0 . 6 , which demonstrates the adaptability and superiority of our method.
[Table]
J C OMPARISON OF RELATED WORK
Table 10 provides a more detailed comparison of Closed Set Recognition, Open Set Recognition (OSR) (Scheirer et al., 2013; Geng et al., 2021), Open World Recognition (OWR) (Bendale & Boult, 2015), and Open-vocabulary Recognition (OVR)
(Radford et al., 2021) from 5 perspectives paradigm, goal, signal, classes type in training, and classes type in testing, respectively.
Contrary to other research, the CLIP-based Open-vocabulary Recognition is unsupervised.
We focus on the model performance on zero-shot inference, without training or fine-tuning on the target dataset.
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
ViT/B32SLIP-ViT/B
CoOp-ViT/B16PE-ViT/B f l o w e r s a q u a t
i c h
o u s e h
o l d m a mm a l s f i s h o u t
d
oo r t h i n g s t
r ee s f r u
i t / v e g e t a b l e s p
e
o p l e s m a ll m a mm a l s v e h
i c
l e s 1 f oo d o u t
d
oo r s c e n e s v e h
i c l e se l e c t r i c a l d e v
i
c
e s i n v e r t e b r a t e s c a r n i v o r
e
s o m n i v o r e s / h e r b i v o r e s r e p t il e s i n s e c t s
[Figure]
Acc-E of Flowers
CLIP-ViT/B16CLIP-ViT/B32DeCLIP-ViT/B32SLIP-ViT/B
CoOp-ViT/B16PE-ViT/B f oo d a q u a t
i
c
h
o u s e h
o l d m a mm a l s f i s h o u t
d
oo r t h i n g s t
r ee s f r u
i t / v e g e t a b l e s p
e
o p l e s m a ll m a mm a l s v e h
i c
l e s 1 f l o w e r s o u t
d
oo r s c e n e s v e h
i c l e se l e c t r i c a l d e v
i
c
e s i n v e r t e b r a t e s c a r n i v o r
e
s o m n i v o r e s / h e r b i v o r e s r e p t il e s i n s e c t s
Acc-E of Food
CLIP-
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
h
o u s e h
o l d a q u a t
i c e l e c t r i c a l d e v
i
c
e s m a mm a l s f
i s h
o u t d oo r t h i n g s t
r ee s f oo d p e
o p l e s m a ll m a mm a l s v e h
i c
l e s 1 f l o w e r s o u t
d
oo r s c e n e s v e h
i c l e sf r u i t / v e g e t a b l e
s i n v e r t e b r a t e s c a r n i v o r
e s o m n i v o r e s / h e r b i v o r e s r e p t il e
[Figure]
i n s e
[Figure]
i n s e c t s
[Figure]
q
[Figure]
t r i c a l
[Figure]
v i c e s m a mm a l s f
i s h
o u t d oo r t h i n g s t
r ee s f oo d p e
o p l e s m a ll m a mm a l s v e h
i c
l e s 1 f l o w e r s o u t
d
oo r s c e n e s v e h
i c l e sf r u i t / v e g e t a b l e
s i n v e r t e b r a t e s c a r n i v o r
e s o m n i v o r e s / h e r b i v o
r e s
[Table]
[Figure]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
i
[Figure]
i
[Figure]
e t
[Figure]
e
[Figure]
i
[Figure]
e r t e
[Figure]
r a t e s i n
[Figure]
e
[Figure]
m n i v
[Figure]
r e
[Figure]
e
[Figure]
i
[Figure] [Figure]
o w e r
[Figure]
o u t
[Figure]
r t h i n g s v e h
i c l e
[Figure] [Figure]
r u i t
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
r t h i n
[Figure]
f
[Figure]
i
[Figure]
t
[Figure]
f r u
[Figure]
[Table]
ll m
[Table]
mm a l s f oo d i n v e r t e
[Table]
r a t e s r e p t il e s v e h
i c
l e s 1 f l o w e r s o u t
d
oo r t h i n g s
v e h
i c
[Table]
f r u i t / v e g e t a
[Table]
l e s m a
[Table]
a l s i n s e c t s o u t
d
oo r s c e n e s p
e
o p l e h
o u s e h
o l d
[Figure]
[Table]
[Figure]
[Table]
l e h
o u s e
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
i v o r
e
s f
i s h c a r n i v o r
e s s m a ll m a mm a l s f oo d i n v e r t e b r a t e s r e p t il e s t
r ee s f l o w e r
s o u t d oo r t h i n g s v e h
i c l e sf r u i t / v e g e t a b l e s m a mm a l s i n s e c t s o u t
d
oo r s c e n e s p
e
o p l e h
o u s e h
o l d
[Figure]
[Table]
/B32SLIP-ViT/B
CoOp-ViT/B16PE-ViT/B f l o w e r s a q u a t
i c h
o u s e h
o l d m a mm a l s f i s h o u t
d
oo r t h i n g s t
r ee s f r u
i t / v e g e t a b l e s p
e
o p l e s m a ll m a mm a l s v e h
i c
l e s 1 f oo d o u t
d
oo r s c e n e s v e h
i c l e se l e c t r i c a l d e v
i
c
e s i n v e r t e b r a t e s c a r n i v o r
e
s o m n i v o r e s / h e r b i v o r e s r e p t il e s i n s e c t s
[Table]
Acc-S of Flowers
CLIP-ViT/B16CLIP-ViT/B32DeCLIP-ViT/B32SLIP-ViT/B
CoOp-ViT/B16PE-ViT/B f oo d a q u a t
i
c
h
o u s e h
o l d m a mm a l s f i s h o u t
d
oo r t h i n g s t
r ee s f r u
i t / v e g e t a b l e s p
e
o p l e s m a ll m a mm a l s v e h
i c
l e s 1 f l o w e r s o u t
d
oo r s c e n e s v e h
i c l e se l e c t r i c a l d e v
i
c
e s i n v e r t e b r a t e s c a r n i v o r
e
s o m n i v o r e s / h e r b i v o r e s r e p t il e s i n s e c t s
[Table]
Acc-S of Food
CLIP
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
h
o u s e h
o l d
[Figure]
q u a t i c e l e c t r i c a l d e v i
c e s m a mm a l s f
i s h
o u t d oo r t h i n g s t
r ee s f oo d p e
o p l e s m a ll m a mm a l s v e h
i c
l e s 1 f l o w e r s o u t
d
oo r s c e n e s v e h
i c l e sf r u i t / v e g e t a b l e
s i n v e r t e b r a t e s c a r n i v o r
e s o m n i v o r e s / h e r b i v o r e s r e p t il e s i n
[Figure]
[Figure]
i n s e c t s a q u a t
i c e l e
[Figure]
t r i c a l d e v
i
c
e s m a mm a l s f
i s h
o u t d oo r t h i n g s t
r ee s f oo d p e
o p l e s m a ll m a mm a l s v e h
i c
l e s 1 f l o w e r s o u t
d
oo r s c e n e s v e h
i c l e sf r u i t / v e g e t a b l e
s i n v e r t e b r a t e s c a r n i v o r
e s o m n i v o r e s / h e r b i v o r
[Table]
c a r n i v o r e s a q u a t
i c e l e c t r i c a l d e v
i
c
e s m a mm a l s f
i s h
o u t
d
[Table]
r t h i n g s t r ee s f
[Table]
d p e o p l e
[Table]
n
[Table]
s
[Table]
l
[Table]
s
[Table]
e r s
[Table]
n e s
[Table]
e h
[Table]
c
[Table]
e sf r u i t / v e g e t a b l e
s i n v e r t e b r a t
e s i n s e c t
[Figure]
e
[Figure]
i
[Figure]
e
[Figure]
i
[Figure]
s
[Figure]
i c l e
[Figure]
f r u i t / v e g e t a b l e
s i n v e r t e b r a t
e s i n s e c t s o m n i v o r e s / h e r b i v o r e s
[Figure]
e
[Figure]
t il
[Figure]
r
[Figure]
i
[Figure]
t r i c a l
[Figure]
e v i
[Figure]
s m a
[Figure]
a
[Figure]
s f
i s h c a r n i v o r e s t r
[Figure]
s f oo d p e o
[Figure]
s m a
[Figure]
m a
[Figure]
a l s v e h i c l e s 1 f l o
w e r s o u t
d oo r t h i n g s v e h
i c l e sf
[Table]
[Figure]
[Table]
c l e s 1 f l o w e r s o u t
d
oo r t h i n g s v e h
i c l e sf r u i t / v e g e t a b l e s m a mm a l s i n s e c t s o u t
d
oo r s c e n e s r e p t il e s h
o u s e h
o l d
[Figure]
i n
[Figure]
e r t
[Figure]
t
[Figure]
e s
[Figure]
l
[Figure]
w e
[Figure]
t
[Figure]
i
[Figure]
s v e h
i c l e sf r u i t / v e g e t a b l e s m
[Figure]
a l s i n s e c t
[Figure]
o u t d
[Figure]
r s
[Figure]
e n e s r e p t il e s h
o
u
s e h o l
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
s h c a r n i v o r
e
s s m a ll m a mm a l s f oo d i n v e r t e b r a t e s r e p t il e s t
r ee s f l o w e r
s o u t d oo r t h i n g s v e h
i c l e sf r u i t / v e g e t a b l e s m a mm a l s i n s e c t s o u t
d
oo r s c e n e s p
e
o p l e h
o u s e h
o l d
[Figure] [Figure]
A d v e r s a r i a l n o n - t a r g
[Figure]
t v o c a
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]