L EARNING TO P ROMPT FOR V ISION -L ANGUAGE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Vision-language pre
-training has recently emerged as a promising alternative for representation learning.
It shifts from the tradition of using images and discrete labels for learning a ﬁxed set of weights, seen as visual concepts, to aligning images and raw text for two separate encoders.
Such a paradigm beneﬁts from a broader source of supervision and allows zero-shot transfer to downstream tasks since visual concepts can be diametrically generated from natural language, known as prompt.
In this paper, we identify that a major challenge of deploying such models in practice is prompt engineering.
This is because designing a proper prompt, especially for context words surrounding a class name, requires domain expertise and typically takes a signiﬁcant amount of time for words tuning since a slight change in wording could have a huge impact on performance.
Moreover, different downstream tasks require speciﬁc designs, further hampering the efﬁciency of deployment.
To overcome this challenge, we propose a novel approach named context optimization (CoOp) .
The main idea is to model context in prompts using continuous representations and perform end-to-end learning from data while keeping the pre-trained parameters ﬁxed.
In this way, the design of task-relevant prompts can be fully automated.
Experiments on 11 datasets show that CoOp effectively turns pre-trained vision-language models into data-efﬁcient visual learners, requiring as few as one or two shots to beat hand-crafted prompts with a decent margin and able to gain signiﬁcant improvements when using more shots (e.g., at 16 shots the average gain is around 17% with the highest reaching over 50%).
CoOp also exhibits strong robustness to distribution shift.
I NTRODUCTION
The traditional approach for visual representation learning is to train vision models to predict for a ﬁxed set of object categories using discrete labels (He et al., 2016; Dosovitskiy et al., 2021).
However, this approach limits visual recognition systems to closed-set visual concepts deﬁned during training, making them unable to handle new categories once deployed in target environments, since additional data are required for learning a new classiﬁer.
Recently, vision-language pre-training such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) has emerged as a promising alternative.
The main idea is to align images and raw text using two separate encoders—one for each modality.
Through large-scale pre-training, vision-language models are allowed to learn open-set visual concepts and can readily be transferred to downstream tasks.
In particular, for each new classiﬁcation task, one can synthesize the classiﬁcation weights by feeding natural language describing classes of interest to the text encoder, and compare them with image features produced by the image encoder.
We observe that for pre-trained vision-language models, the text input, known as prompt, plays a key role in downstream datasets.
However, identifying the right prompt is a non-trivial task, which often takes a signiﬁcant amount of time for words tuning—since a slight change in wording could make a huge difference in performance.
For instance, for Caltech101 (Figure 1(a), 2nd vs. 3rd prompt), adding “a” before the class token brings more than 5% increase in accuracy.
Moreover, prompt engineering also requires expertise about the task and ideally the language model’s underlying mechanism.
This is exempliﬁed in Figure 1(b-d) where adding task-relevant context can lead to signiﬁcant improvements, i.e., “ﬂower” for Flowers102, “texture” for DTD and “satellite” for
EuroSAT.
Tuning the sentence structure could bring further improvements, e.g., putting “a type of ﬂower” after the class token for Flowers102, keeping only “texture” in the context for DTD, and
Describable Textures (DTD)
EuroSAT a [CLASS].
a photo of [CLASS]. a photo of a [CLASS].
[V] 1 [V] 2 …
[V] M [CLASS].
Flowers a photo of a [CLASS].
a flower photo of a [CLASS].
a photo of a [CLASS], a type of flower.
[V] 1 [V] 2 …
[V] M [CLASS].
a photo of a [CLASS].
a photo of a [CLASS] texture.
[CLASS] texture.
[V] 1 [V] 2 …
[V] M [CLASS].
a photo of a [CLASS].
a satellite photo of [CLASS]. a centered satellite photo of [CLASS].
[V] 1 [V] 2 …
[V] M [CLASS].
Caltech
Prompt Prompt
Prompt Prompt
Accuracy Accuracy
Accuracy Accuracy (
[Figure] [Figure]
[Figure]
adding “centered” before “satellite photo” for EuroSAT.
However, even with extensive tuning, the resulting prompts are by no means guaranteed to be optimal for these downstream tasks.
Inspired by recent prompt learning research in NLP (Shin et al., 2020; Jiang et al., 2020; Zhong et al., 2021), we propose context optimization (CoOp) 1 to automate prompt engineering to allow more efﬁcient and task-speciﬁc transfer for pre-trained vision-language models.
Speciﬁcally, we model a prompt’s context using continuous representations which are essentially initialized with random vectors with the same dimension as word embeddings (see Figure 2).
The context could be shared among all classes or designed to be class-speciﬁc.
During training, we simply minimize the prediction error using the cross-entropy loss with respect to the learnable context vectors, while keeping the pre-trained parameters ﬁxed.
The gradients can be back-propagated all the way through the text encoder, distilling the rich knowledge encoded in the parameters for learning task-relevant context.
To demonstrate the effectiveness of CoOp, we benchmark on 11 datasets, which cover a diverse set of visual recognition tasks including classiﬁcation on generic objects, scenes, actions and ﬁne-grained categories, as well as specialized tasks like recognizing textures and satellite imagery.
The results show that CoOp can effectively turn pre-trained vision-language models into data-efﬁcient visual learners, requiring as few as one or two shots to beat hand-crafted prompts with a decent margin.
The performance can also be further boosted by using more shots, e.g., at 16 shots the margin over hand-crafted prompts averages at around 17% and reaches over 50% for the highest.
CoOp also outperforms the linear probe alternative known as a strong few-shot learning baseline (Tian et al., 2020), and crucially, demonstrates much stronger robustness to distribution shift.
Extensive analysis is also conducted to offer a comprehensive picture on how to apply CoOp in practice.
The source code for reproducing the experiments will be released to facilitate future research.
M ETHODOLOGY 2. V ISION -L ANGUAGE P RE - TRAINING
We brieﬂy introduce vision-language pre-training with a particular focus on CLIP (Radford et al., 2021).
Our approach is applicable to broader CLIP-like vision-language models.
Models CLIP consists of two encoders, one for images and the other for text.
The image encoder aims to map high-dimensional images into a low-dimensional embedding space.
The architecture of the image encoder can take the form of a CNN like ResNet-50 (He et al., 2016) or a ViT (Dosovitskiy et al., 2021).
On the other hand, the text encoder is built on top of a Transformer (Vaswani et al., 2017) and aims to generate text representations from natural language.
[Table]
+egaNdAdaqIWIoijZ/SK3pwX5935cD4XrRtOPnOGluB8/QImmZ+d</latexit> ... text encoder image encoder text features imagefeatures similarityscores <latexit sha1_base64="i4zpmuRm7l6oli4lXwkT2OQ8Jsk=">AAACFXicbVC7TgJBFJ31ifhCLW02ggkV2aVQSxIbS0zkkcCGzM7OwoR5bGbuasiGn7Cx0F+xM7bW/omlA2wh4ElucnLOvbn3njDhzIDnfTsbm1vbO7uFveL+weHRcenktG1UqgltEcWV7obYUM4kbQEDTruJpliEnHbC8e3M7zxSbZiSDzBJaCDwULKYEQxW6lb6o0iBqQxKZa/mzeGuEz8nZZSjOSj99CNFUkElEI6N6fleAkGGNTDC6bTYTw1NMBnjIe1ZKrGgJsjm907dS6tEbqy0LQnuXP07kWFhzESEtlNgGJlVbyb+5/VSiG+CjMkkBSrJYlGccheUO3vejZimBPjEEkw0s7e6ZIQ1JmAjWtoSiqUfMpFyYFo9TYs2Kn81mHXSrtf8q5p3Xy83qnloBXSOLlAV+egaNdAdaqIWIoijZ/SK3pwX5935cD4XrRtOPnOGluB8/QImmZ+d</latexit> ...
[V
[Table]
[V
[Table]
<
[Table]
sha
[Table]
=
[Table]
=
[Table]
[Figure]
Speciﬁcally, given a sequence of words (tokens), such as “a photo of a dog.”, CLIP ﬁrst converts each one of the token (including punctuation) into a lower-cased byte pair encoding (BPE) representation (Sennrich et al., 2016), which is essentially a unique numeric ID.
The vocabulary size in CLIP is 49,152.
To facilitate minibatch processing, each text sequence is encompassed with the [SOS] and [EOS] tokens and capped at a ﬁxed length of 77.
After that, the IDs are mapped to 512-D word embedding vectors, which are then passed on to the Transformer.
Finally, the features at the [EOS] token position are layer normalized and further processed by a linear projection layer.
Training CLIP is trained to align the two embedding spaces learned for images and text respectively.
Speciﬁcally, the learning objective is formulated as a contrastive loss.
Given a batch of image-text pairs, CLIP maximizes the cosine similarity for matched pairs while minimizes the cosine similarity for all other unmatched pairs.
To learn diverse visual concepts that are more transferable to downstream tasks, CLIP’s team collects a large training dataset consisting of 400 million image-text pairs.
Zero-Shot Inference Since CLIP is pre-trained to predict whether an image matches a textual description, it naturally ﬁts zero-shot recognition.
This is achieved by comparing image features with the classiﬁcation weights synthesized by the text encoder, which takes as input textual descriptions specifying classes of interest.
Formally, let f be image features extracted by the image encoder for an image x and { w i } Ki =1 a set of weight vectors generated by the text encoder.
K denotes the number of classes and each w i is derived from a prompt that could have the form of “a photo of a [CLASS].”
where the class token is replaced by the speciﬁc class name, such as “cat”, “dog” or “car”.
The prediction probability is then computed as
[Equation]
where τ is a temperature parameter learned by CLIP and < · , · > denotes cosine similarity.
2. C ONTEXT O PTIMIZATION
We propose context optimization (CoOp), which avoids manual prompt tuning by modeling context words with continuous vectors that are end-to-end learned from data.
An overview is shown in
Figure 2.
Speciﬁcally, the prompt given to the text encoder g ( · ) is designed with the following form,
[Equation]
where each [ V ] m ( m ∈{ 1 , . .
, M } ) is a vector with the same dimension as word embeddings (i.e., 512 for CLIP), and M is a hyperparameter specifying the number of context tokens.
Note that the context here is shared among all classes, which is called uniﬁed context and different from classspeciﬁc context that is introduced later.
By forwarding a prompt t to the text encoder g ( · ) , we can obtain a classiﬁcation weight vector representing a visual concept.
The prediction probability is computed as
[Equation]
where the class token within each prompt t i is replaced by the corresponding word embedding vector(s) of the i -th class name.
Training is performed to minimize the standard classiﬁcation loss based on the cross-entropy, and the gradients can be back-propagated all the way through the text encoder g ( · ) , making use of the rich knowledge encoded in the parameters to optimize the context.
The design of continuous representations also allows full exploration in the word embedding space, which facilitates the learning of task-relevant context.
Other Variants Other than placing the class token at the end of a sequence as in Equation (2), we can also put it in the middle like
[Equation]
which increases ﬂexibility for learning—theoretically, the prompt is allowed to either ﬁll the latter cells with supplementary descriptions or cut off the sentence earlier by using a termination signal such as full stop.
Another option is to design class-speciﬁc context (CSC) where context vectors are independent to each class, i.e., [ V ] i 1 [ V ] i 2 . . .
[ V ] iM (cid:54) =
[ V ] j 1 [ V ] j 2 . . .
[ V ] jM for i (cid:54) =
j and i, j ∈ { 1 , . .
. , K } .
As an alternative to uniﬁed context, we ﬁnd that CSC is particularly useful for some ﬁne-grained classiﬁcation tasks.
E XPERIMENTS 3. F EW -S HOT L EARNING
Datasets We select 11 publicly available image classiﬁcation datasets used in CLIP: ImageNet (Deng et al., 2009), Caltech101 (Fei-Fei
et al., 2004), OxfordPets (Parkhi et al., 2012), StanfordCars (Krause
et al., 2013), Flowers102 (Nilsback & Zisserman, 2008), Food101 (Bossard et al., 2014), FGVCAircraft (Maji et al., 2013), SUN397 (Xiao et al., 2010), DTD (Cimpoi et al., 2014),
EuroSAT (Helber et al., 2019) and UCF101 (Soomro et al., 2012) (see Appendix A for their statistics).
These datasets constitute a comprehensive benchmark, which covers a diverse set of vision tasks including classiﬁcation on generic objects, scenes, actions and ﬁne-grained categories, as well as specialized tasks like recognizing textures and satellite imagery.
We follow the few-shot evaluation protocol adopted in CLIP (Radford et al., 2021), using 1, 2, 4, 8 and 16 shots for training respectively and deploying models in the full test sets.
The average results over three runs are reported for comparison.
Training Details CoOp has four versions: positioning the class token in the end or middle; uniﬁed context vs. CSC.
Unless otherwise stated, ResNet-50 (He et al., 2016) is used as the image encoder’s backbone and the number of context tokens M is set to 16.
Investigations on other design choices are discussed in Section 3.3.
All models are built on top of the open-sourced CLIP.
2 CoOp’s context vectors are randomly initialized by drawing from a zero-mean Gaussian distribution with standard deviation equal to 0.02.
Training is done with SGD and an initial learning rate of 0.002, which is decayed by the cosine annealing rule.
The maximum epoch is set to 200 for 16/8 shots, 100 for 4/ shots, and 50 for 1 shot (except for ImageNet where the maximum epoch is ﬁxed to 50).
To mitigate explosive gradients observed in the early training iterations, we use the warmup trick by ﬁxing the learning rate to 1 e − 5 during the ﬁrst epoch.
Baseline Methods We compare CoOp with two baseline methods.
The ﬁrst is zero-shot CLIP, which is based on hand-crafted prompts.
We follow the guideline of prompt engineering introduced by Radford et al.
For generic objects and scenes, “a photo of a [CLASS].”
is adopted.
For ﬁne-grained categories, task-relevant context is added like “a type of pet” for OxfordPets and “a type of food” for Food101.
When it comes to specialized tasks such as recognizing textures in
DTD, the prompt is customized as “[CLASS] texture.”
where the class names are adjectives like “bubbly” and “dotted”.
See Appendix A for second
As suggested by Radford et al.
(2021) and a recent study on few-shot learning (Tian et al., 2020), training a linear classiﬁer on top of high-quality pre-trained models’ features (like CLIP) can easily achieve performance that is on a par with that of state-of-the-art few-shot learning methods, which are often much more sophisticated.
We follow the same training method used by Radford et al.
(2021) to train linear probe CLIP.
Comparison with Hand-Crafted Prompts Figure 3 summarizes the results.
Our default model is CLIP+CoOp with the class token positioned in the end.
The two different ways of positioning the class token achieve similar performance as their curves highly overlap.
From the average performance displayed in the top-left corner, we observe that CLIP+CoOp is a strong few-shot learner, requiring only two shots on average to obtain a decent margin over zero-shot CLIP.
Given 16 shots for training, the average gap brought by CoOp can be further increased to around 17%.
Figure 4 ranks the absolute improvements obtained by CoOp at 16 shots over hand-crafted prompts.
Huge improvements are observed on specialized tasks namely EuroSAT and DTD where the increase in performance reaches over 50% and 20% respectively.
The jumps in performance are also signiﬁcant (those more than 10%) on most ﬁne-grained datasets including Flowers102, StanfordCars and FGVCAircraft, as well as on scene and action recognition datasets (SUN397 & UCF101).
Since ImageNet is a challenging dataset that contains 1,000 classes, the 5.05% improvement is also noteworthy.
In contrast, the increases on the two ﬁne-grained datasets, OxfordPets and Food101, are less appealing.
By digging into CLIP+CoOp’s curves on these two datasets in Figure 3, we ﬁnd there is a loss of momentum in performance improvements even with more shots used, seemingly an overﬁtting problem.
A potential solution is to impose higher regularizations like increasing the weight decay.
Nonetheless, the overall results are strong enough to serve as evidence of CoOp’s capability of learning task-relevant prompts in a data-efﬁcient way.
[Figure]
Comparison with Linear Probe CLIP In terms of the overall performance (Figure 3, topleft), CLIP+CoOp demonstrates clear advantages over linear probe CLIP.
The latter requires 4 shots on average to match the zero-shot’s performance while CoOp’s average gains at 4 shots are already more than 10%.
It is also clear that the gaps in the extreme low-data regime such as one or two shots are much larger, suggesting that CoOp is much more effective than learning a linear classiﬁer from scratch for fewshot learning.
We also observe that linear probe
CLIP is comparable to CLIP+CoOp on the two specialized tasks (DTD & EuroSAT) as well as on a couple of ﬁne-grained datasets (Flowers102 & FGVCAircraft)—this is not too surprising as the pre-trained CLIP space has been proved powerful, making the linear probe model a strong competitor.
Nevertheless, CoOp’s CSC version can beat linear probe CLIP on the aforementioned datasets, and moreover, shows much better potential when more shots become available.
Uniﬁed vs. Class-Speciﬁc Context On average, using uniﬁed context leads to better performance.
In terms of when to apply CSC and when not to, we have the following suggestions.
For generic objects (ImageNet & Caltech101), scenes (SUN397) and actions (UCF101), using uniﬁed context is clearly better.
Uniﬁed context also works better on some ﬁne-grained datasets including OxfordPets and Food101, but on others like StanfordCars, Flowers102 and FGVCAircraft the CSC version is preferred.
CSC also yields better performance on the two specialized tasks, DTD and EuroSAT, at 16 shots in particular.
However, CSC mostly underperforms uniﬁed context in challenging low-data scenarios (fewer than 8 shots), which makes sense because CSC has more parameters than uniﬁed context and needs more data for training.
3. R OBUSTNESS TO D ISTRIBUTION S HIFT
Since CoOp requires training on a speciﬁc data distribution, it risks learning spurious correlations that are detrimental to generalization in unseen distributions (domains), as suggested in recent studies (Taori et al., 2020;
Zhou et al., 2021).
On the contrary, zero-shot CLIP is not tied to a speciﬁc data distribution and has exhibited strong robustness to distribution shift (Radford et al., 2021).
In this section, we aim to unveil how robust CoOp is to distribution shift, in comparison to zero-shot
CLIP and the linear probe model.
[Table] [Table]
[Table]
Datasets The source dataset is ImageNet. The target datasets are ImageNetV2 (Recht et al., 2019),
ImageNet-Sketch (Wang et al., 2019), ImageNet-A (Hendrycks et al., 2021b) and ImageNetR (Hendrycks et al., 2021a), all of which have compatible class names with ImageNet allowing seamless transfer for the prompts learned by CoOp.
ImageNetV2 is a reproduced test set using different sources while following ImageNet’s data collection process.
ImageNet-Sketch contains sketch images belonging to the same 1,000 ImageNet classes.
Both ImageNet-A and -R containclasses derived from a subset of ImageNet’s 1,000 classes.
The former consists of real-world adversarially ﬁltered images that cause current ImageNet classiﬁers to produce low results, whereas the latter features a rendition of the ImageNet classes in diverse image styles such as paintings, cartoons and sculptures.
Results Table 1 summarizes the results.
It is surprising that CLIP+CoOp exhibits stronger robustness than zero-shot CLIP to distribution shift, despite exposure to the source dataset.
This suggests that the learned prompts are also generalizable.
Moreover, it is interesting to see that using fewer context tokens leads to better robustness.
More results with different vision backbones are provided in Appendix B.1 where the conclusion remains the same.
In contrast, linear probe CLIP obtains much worse results on these target datasets, exposing its weakness in domain generalization.
3. F URTHER A NALYSIS
Comparison with Prompt Ensembling Radford et al.
(2021) have suggested that additional improvements can be obtained by ensembling over multiple zero-shot classiﬁers generated using different hand-crafted prompts, such as “a photo of the large [CLASS].”
, “a bad photo of the [CLASS].”
and “a origami [CLASS].”, which reﬂect a different scale, view and abstraction respectively for an image.
We are interested to know whether the prompts learned by CoOp can still maintain advantages when compared with prompt ensembling.
For fair comparison, we use the select prompts from Radford et al.
(2021), which have been extensively tuned on ImageNet, to construct the ensemble classiﬁer.
Table 2 presents the results of prompt engineering (i.e., using a single hand-crafted prompt), prompt ensembling and CoOp, conﬁrming that CoOp is still the best performing method.
Additional results are provided in Appendix B.2 to show that CoOp also beats prompt ensembling when more advanced vision backbones are used.
Given the potential of prompt ensembling, future work could investigate how to improve CoOp from the ensembling perspective.
Context Length How many context tokens should be used?
And is it better to have more context tokens?
The results in Section 3.2 suggest having shorter context length beneﬁts domain generalization.
Here we study this hyperparameter for source datasets.
Speciﬁcally, we repeat experiments on the 11 datasets by varying the context length from 4 to 8 to 16.
The average results are shown in Figure 5(a), which indicate that having more context tokens leads to better performance and that positioning the class token in the middle gains more momentum with longer context length.
To sum
[Figure]
up, there is no golden rule for selecting perfect context length since one needs to balance between performance and robustness to distribution shift.
See Appendix B.3 for the dataset-speciﬁc results.
Vision Backbones Figure 5(b) summarizes the results on the 11 datasets using a variety of vision backbones covering both CNNs and ViTs.
The results are expected: the more advanced the backbone, the better the performance.
The gap between CoOp and hand-crafted prompts is signiﬁcant across all architectures.
See Appendix B.4 for the dataset-speciﬁc results.
Initialization We compare random initialization with manual initialization.
The latter uses the embeddings of “a photo of a” to initialize the context vectors for the 11 datasets.
For fair comparison, we also set the context length to 4 when using random initialization.
Table 3 suggests a “good” initialization only brings a small improvement.
Though further tuning of the initialization words might help, in practice we suggest using the simple random initialization method.
Interpreting the Learned Prompts is difﬁcult because the context vectors are optimized in a continuous space.
We resort to an indirect way by searching within the vocabulary for words that are closest to the learned vectors based on the Euclidean distance.
Note that CLIP (Radford et al., 2021) uses the BPE representation (Sennrich et al., 2016) for tokenization, so the vocabulary includes subwords that frequently appear in text, such as “hu” (subsumed by many words like “hug” and “human”).
Table 4 shows the searched results on some datasets.
We observe that a few words are somewhat relevant to the tasks, such as “enjoyed” for Food101, “ﬂuffy” and “paw” for OxfordPets, and “pretty” for DTD.
But when connecting all the nearest words together, the prompts do not make much sense.
We also observe that when using manual initialization (like “a photo of a”), the nearest words for the converged vectors are mostly the ones used for initialization.
We conjecture that the learned vectors might encode meanings that are beyond the existing vocabulary.
Overall, we are unable to draw any ﬁrm conclusion based on the observations because using nearest words to interpret the learned prompts could be inaccurate—the semantics of the vectors is not necessarily correlated with the nearest words.
R ELATED W ORK
Vision-Language Models have recently demonstrated great potential in learning generic visual representations and allowing zero-shot transfer to a variety of downstream classiﬁcation tasks (Radford et al., 2021;
Jia et al., 2021; Zhang et al., 2020).
To our knowledge, the recent developments in vision-language learning, particularly CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), are largely driven by advances in the following three areas: i) text representation learning with Transformers (Vaswani et al., 2017), ii) large-minibatch contrastive representation learning (Chen et al., 2020; He et al., 2020; H´enaff et al., 2020), and iii) web-scale training datasets—CLIP beneﬁts from 400 million curated image-text pairs while ALIGN exploits 1.8 billion noisy image-text pairs.
The idea of mapping images and text onto a common embedding space has been studied since nearly a decade ago (Socher et al., 2013; Frome et al., 2013; Elhoseiny et al., 2013), but with drastically
[Table]
different technologies.
For text features extraction, early work has mainly utilized pre-trained word vectors (Socher et al., 2013; Frome et al., 2013) or the hand-crafted TF-IDF features (Elhoseiny et al., 2013; Lei Ba et al., 2015).
Matching images and text features has been formulated as metric learning (Frome et al., 2013), multi-label classiﬁcation (Joulin et al., 2016; Gomez et al., 2017), n-gram language learning (Li et al., 2017), and the recently proposed captioning (Desai & Johnson, 2021).
Our work is orthogonal to recent research in vision-language models, aiming to facilitate the deployment of such models in downstream datasets.
Prompt Learning in NLP Knowledge probing for large pre-trained language models, formally deﬁned by Petroni et al.
(2019) as “ﬁll-in-the-blank” cloze tests, has recently sparked interest in prompt learning research in NLP (Shin et al., 2020; Jiang et al., 2020; Li & Liang, 2021;
Zhong et al., 2021; Lester et al., 2021; Gao et al., 2020).
The basic idea of knowledge probing is to induce pre-trained language models to generate answers given cloze-style prompts, which can beneﬁt a number of downstream tasks, such as sentiment analysis.
Jiang et al.
(2020) propose to generate candidate prompts through text mining and paraphrasing, and identify the optimal ones that give the highest training accuracy.
Shin et al.
(2020) introduce a gradient-based approach, which searches for tokens with the largest gradient changes in the label likelihood.
Most related to our work are continuous prompt learning methods (Zhong et al., 2021; Li & Liang, 2021; Lester et al., 2021) which optimize continuous vectors in the word embedding space.
A drawback of such methods compared to searching discrete tokens is the lack of a clear way to visualize what “words” are learned for the vectors.
We refer readers to Liu et al.
(2021) for a comprehensive survey in the topic of prompt learning in NLP.
C ONCLUSION
We present CoOp, a differentiable approach that focuses on continuous prompt learning to facilitate the deployment of pre-trained vision-language models in downstream datasets.
The results on the 11 datasets serve as strong evidence of CoOp’s effectiveness in data-efﬁcient learning.
The learned prompts are proved much more task-relevant than hand-crafted prompts reﬂected by the huge improvements in performance, as well as stronger in robustness to distribution shift.
In terms of limitation, CoOp requires explicit access to the pre-trained model parameters, which might be unavailable when only the APIs of pre-trained models are provided.
An interesting future direction is thus to investigate “black-box” prompt learning.
A PPENDIX A D ATASETS D ETAILS
The detailed statistics of the 11 datasets, as well as the four variants of ImageNet, are shown in
Table 5.
The hand-crafted prompts used for zero-shot CLIP are also detailed in the table.
For
Caltech101, the “BACKGROUND Google” and “Faces easy” classes are discarded.
For the video dataset, UCF101, the middle frame of each video is used as input to the image encoder.
[Table]
/A 10, “a photo of a [CLASS].”
ImageNet-Sketch 1,
N/A
N/A 50, “a photo of a [CLASS].”
ImageNet-A
N/A
N/A 7, “a photo of a [CLASS].”
ImageNet-R
N/A
N/A 30, “a photo of a [CLASS].”
B A DDITIONAL R ESULTS B. R OBUSTNESS E XPERIMENTS
In addition to ResNet-50, we further experiment with more advanced architectures including
ResNet-101, ViT-B/32 and ViT-B/16, all of which have pre-trained weights available from CLIP’s GitHub repository.
The results are shown in Table 6 where we can draw the same conclusion as the main paper: CoOp offers stronger robustness than hand-crafted prompts and using fewer context tokens beneﬁts domain generalization
. B. P ROMPT E NGINEERING , P ROMPT E NSEMBLING AND C O O P
Table 7 provides more comprehensive comparisons covering a variety of vision backbones.
The observations are similar to those discussed in the main paper: prompt ensembling is clearly better than prompt engineering; and CoOp demonstrates consistent advantages over prompt ensembling.
B. C ONTEXT L ENGTH
Figure 6 shows detailed results of using different context lengths for CoOp on each of the 11 datasets.
The average performance, displayed in the top-left corner, suggests that using more context tokens is better.
There are three exceptions: on ImageNet, OxfordPets, and Food101, the performance is saturated and the improvements diminish when the context length is increased.
As discussed in the main paper, selecting a proper length needs to balance between performance on source datasets and robustness to distribution shift in unseen domains.
We suggest practitioners use a validation set to identify the optimal context length for their applications.
[Table]
[Table]
B. V ISION B ACKBONES
Figure 7 provides the detailed per-dataset results for various vision backbones.
The more advanced the backbone, the better the performance.