Prompt Learning with Optimal Transport for Vision-Language Models
Anonymous Author(s)
AffiliationAddress email
Abstract
With the increasing attention to large vision-language models such as CLIP, therehas been a significant amount of effort dedicated to building efficient prompts.
Unlike conventional methods of only learning one single prompt we proposeto learn multiple comprehensive prompts to describe diverse characteristics ofcategories such as intrinsic attributes or extrinsic contexts.
However, directlymatching each prompt to the same visual feature is problematic, as it pushes theprompts to converge to one point.
To solve this problem, we propose to applyoptimal transport to match the vision and text modalities.
Specifically, we firstmodel images and the categories with visual and textual feature sets.
Then, weapply a two-stage optimization strategy to learn the prompts.
In the inner loop, weoptimize the optimal transport distance to align visual features and prompts by the
Sinkhorn algorithm, while in the outer loop, we learn the prompts by this distancefrom the supervised data.
Extensive experiments are conducted on the few-shotrecognition task and the improvement demonstrates the superiority of our method
Introduction
In the past few years, large-scale vision-language pre-trained (VLP) models, such as CLIP [39],
ALIGN [17], and BLIP [23] have achieved remarkable success in open-world visual concept learning.
These methods have brought new light but also pose a new question: how to efficiently adapt theknowledge from pretraining to the downstream tasks since these models are typical of massive sizeswhich are not feasible for normal users to re-train.
Brambling
A bird that lives in winter wood
A bird with dark fan-tail
A bird with orange and black texture
A bird with black crown and eye
“Brambling”) One of the conventionalparadigms of utilizing pretrainedknowledge is “pre-training,fine-tuning”, which fixes thearchitecture of the pre-trainedneural network and tunes itsparameters using task-specificobjective functions.
Beyondfine-tuning the parameters,many recent methods [63, 64]introduce the concept of promptlearning from the field of NLP to the vision domain and achieve striking performance gain for thefew-shot visual classification.
They fix the model parameters and instead learn suitable promptsby turning a template sentence into a set of learnable vectors.
Then, these prompts are learned byminimizing the distance between the visual features and prompt-based language features.
Despite significant improvements over manual prompts, learning only a sentence is intuitivelyinsufficient to represent a class.
One class can be described by many intrinsic characteristics and even extrinsic context relations.
Thus, for one object, we may have multiple prompt candidateswhich focus on different attributes.
As shown in Figure 1, we can describe the class “Brambling” indifferent views: such as the color of the wing, the color of the crown and eyes, the shape and color ofthe tail, and even the living environment information.
It motivates us to learn multiple prompts tocomprehensively represent the class and thus facilitate classification.
The most natural solution is to directly learn multiple prompts by respectively matching each promptwith the visual features.
However, it is the same as matching the mean of prompt features and thevisual features.
This solution is problematic since all prompts are encouraged to be closer to one singlepoint and thus tend to learn the same characteristics.
It contradicts our purpose to learn comprehensiveprompts.
To solve this problem, we tested adding some constraints to push away the prompt fromeach other, but found that this solution still fails to learn representative and comprehensive prompts.
This solution treats the visual representation as one single point, and such a unified view of visualfeatures ignores the fact that different prompts may only focus on one or a subset of characteristics.
To address this problem, in this paper, we propose Prompt Learning with Optimal Transport (PLOT),which applies optimal transport (OT) to align the local visual features and multiple textual prompts.
Optimal transport can calculate the distance between two distributions under the form of multiplesampling.
In our prompt learning framework, we formulate local visual features and multiple promptsas the samplings of two discrete distributions and use OT to encourage fine-grained cross-modalmatching.
Specifically, to obtain the local visual features with different semantic clues, we extract allfeature maps as the visual representation instead of the single global representation.
Fortunately, wecan easily obtain the visual feature maps from the visual encoder of CLIP by using all outputs of themulti-head self-attention layer [42].
Then the problem comes down to how to calculate the distancebetween two feature sets.
We solve this problem by introducing the optimal transport theory [51] and formulate the feature setsas a discrete probability distribution where each feature has an equal probability value.
Furthermore,to reduce the computational cost and avoid the extra model parameters, we learn the prompts witha two-stage optimization strategy.
At the first stage in the inner loop, we fix both visual and textfeatures and optimize the optimal transport problem by a fast Sinkhorn distances algorithm
Then,in the outer loop, we fix all parameters of optimal transport and back-propagate the gradient to learnthe prompts with different characteristics.
Compared with conventional distance (such as Euclideandistance of mean features), optimal transport can align different visual features for each local prompt,which is more robust to the visual misalignment and tolerates well feature shift [44].
It is because OTlearns an adaptive transport plan to align features, which achieves fine-grained matching across twomodalities.
We conduct experiments on 11 datasets following the standard setting of CLIP [39] and CoOp [63] to evaluate our method.
These experiments span the visual classification of generic objects,scenes, actions, fine-grained categories, and so on.
The significant result improvement demonstratesthat PLOT can effectively learn representative and comprehensive prompts.
Related Work
Optimal Transport The Optimal Transport [30] is initially introduced to solve the problem of howto reduce the cost when moving several items simultaneously.
Recently, OT theory has drawn wideattention in the machine learning and computer vision community by comparing distributions readilyavailable to them under the form of feature sets [37].
Due to the brilliant property of distributionmatching, OT has been applied in many theoretic and application tasks including generative models [1,45, 60], structural matching [4, 57, 61, 56] (e.g. sequence matching [4] and graph matching [56]),and other distribution-based tasks (such as clustering [22], distribution estimation [2], and causaldiscovery [50]).
In this paper, we use OT to align the features of vision and language modalitieswhich represents the data structure by learning an adaptive transport plan [44].
Vision-Language Pre-trained Models Vision-Language Pre-trained (VLP) models aim to explorethe semantic correspondence between the vision and language modalities through large-scale pretraining.
Recently, VLP models have achieved an exciting performance improvement in the zero-shotand few-shot visual recognition [39, 10, 63, 64, 59], which shows the great potential to promoteopen-world visual understanding with the help of language.
One key part of learning VLP models isthe self-supervised learning objective on two modalities.
The popular VLP objectives can be dividedinto reconstruction [25, 15, 8, 20], contrastive matching [39, 17, 16], or the combination of bothtwo [24, 54, 19].
Besides, recent progress in the field of VLP also benefits a lot from large-scale pair-wised datasets.
For example, CLIP [39] applies 400 million image-text pairs for contrastivelearning, while ALIGN even exploits 1.8 billion data pairs.
Beyond recognition, these VLP modelsalso show great potential for other downstream applications, such as dense prediction [42, 62], imagegeneration [31, 41, 35], and action understanding [53, 48].
Prompt Learning Prompt learning is introduced from the field of NLP to efficiently adapt the largelanguage model to downstream tasks.
Different from the conventional “pre-training, fine-tuning”paradigm which initializes the pre-trained model and tunes the parameters of the network usingdownstream task-specific objective functions, prompt learning applies textual prompt to reformulatethe downstream tasks as the original pretrained task [27, 36].
By the prompt, the domain shift betweenpretrained task and downstream application is reduced and thus the pretrained knowledge can beeasier adapted to downstream tasks.
The concept of prompt learning [36, 40, 38] begins from thesuccess of GPT [40] series.
Early prompt learning methods (such as Petroni et al.
[36] and Pörner etal.
[38]) always manually create templates based on human prior knowledge.
Furthermore, somemining-based methods [18] and gradient-based methods [46] are proposed to automatically search forappropriate templates.
Beyond search in the discrete space, some methods [26, 49, 28] remove theconstraint that the prompts are “words” and instead learn prompts in the continuous embedding space.
Recently, CoOp [63] and its extended version [64] introduce prompt learning into open-world visualunderstanding to adapt the knowledge from the large-scale visual-language pretrained models andachieve great performance improvement on the few-shot visual recognition.
Compared with CoOp,our PLOT method further improves prompt learning by introducing the optimal transport distance tolearn multiple local prompts and achieves fine-grained vision-language matching.
Approach
In this section we will first revisit the baseline method CoOp 3.1, review the preliminaries ofoptimal transport 3.2, and then introduce our proposed PLOT 3.3 to show how we can learn multiplecomprehensive prompts.
3.1 A Revisit of CoOp
CoOp [63] is one of the pioneering methods to learn the prompts for using vision language pretrainedknowledge (such as CLIP [39]) for downstream open-world visual recognition.
Different from CLIPwhich manually designs the prompt templates, CoOp sets a part of context words in the template ascontinuous learnable parameters which can be learned from the few-shot data.
Then the classificationweights can be represented by the distance between the learned prompt and visual feature.
Specifically, given an image x , a visual feature f = f ( x ) is obtained by the visual encoder f of
CLIP.
Then, the textual prompt can be formulated as t k = { vec 1 , vec 2 , . .
, vec L , c k } , where c k isthe word embedding of the class name, { vec l | Ll =1 } are learnable vectors with the same dimension asthe original word embedding and L is the length of context words.
With prompt t k as the input, thetext encoder g outputs the textual feature as g k = g ( t k ) .
The final prediction probability is computedby the matching score as follows:
[Equation]
where sim ( · , · ) denotes a metric function such as cosine similarity, and τ stands for the temperatureof Softmax.
Then we can optimize the parameters of { vec l | Ll =1 } with the cross-entropy loss betweenthe prediction and the labeled target.
3.2 Optimal Transport
Optimal transport (OT) distance is a widely used metric for the comparison of distributions.
Here, weonly focus on the discrete situation which is more related to our framework.
Assuming we have twosets of points (features), the discrete distributions are formulated as:
[Equation]
where u and v are the discrete probability vectors that sum to 1, and δ f is a Dirac delta functionplaced at support point f
in the embedding space.
Then, the total distance of these two distributions
[X X
[Equation]
CLASS
[Equation]
[Figure]
Brambling, House finch, …
[Figure]
OptimalTransportDistance are written as:
[Equation]
We call C the cost matrix in which each point denotes the cost between f m and g n , such as
C m,n = 1 − sim
( f m , g n ) .
While the T is called the transport plan, which is learned to minimize thetotal distance.
The optimization problem of optimal transport is formulated as:
[Equation]
subject to [Equation] [Equation] As directly optimizing the above objective is always time-consuming, we apply the Sinkhorn distance [6] to use an entropic constraint for fast optimization.
The optimization problem with a
Lagrange multiplier of the entropy constraint is:
[Equation]
subject to
[Equation]
(5) where h ( · ) is entropy and λ ≥ 0 is a hyper-parameter.
Then we can have a fast optimization solutionwith a few iterations as:
[Equation]
(6) where t denotes iteration and in each iteration u t = u / (( exp ( − C /λ ) v t − 1 ) and v t =v / (( exp ( − C /λ )
T u t ) , with the initiation v 0 = 1 .
3.3 Prompt Learning with Optimal Transport
In this subsection, we introduce the details of our PLOT, which learns multiple prompts to describedifferent characteristics of the category by minimizing the OT distance.
Specifically, as shown in Figure 2, given an image x , we first feed it to the visual encoder branch of
CLIP.
Apart from the global visual feature f , we can also obtain a set of local features { f m |
Mm
=1 } .
The visual encoder has a multi-head attention pooling layer in which the input is the combination ofthe global feature and a set of local features (feature map) and the output is a tensor with the shape 156 R ( H × W +1)
× C , where H and W is the height and width of feature map and C is the feature dimension.
Therefore, we can obtain M = H × W local features and a global feature.
At the same time, forclass k , we can initialize N local prompts as { t k,n | Nn =1 } with learnable vectors { vec l,n | L,Nl =1 ,n =1 } ,where each is the same as the prompt in CoOp.
With both visual and textual encoders, we can obtainlocal visual features F = { f m | Mm =1 } ∈ R M × C and prompt features G k = { g n | Nn =1 } ∈ R N × C .
In the inner loop, we learn the transport plan T with these fixed support sets F , G k , by minimizingthe following OT distance to push G k to F :
[Equation]
where C = 1 − F T G k denotes that we use the cosine distance between F and G k as the cost matrix.
Then we can obtain the solution of transport plan T ∗ as Eq (6) and the final OT distance d O T ( k ) .
Given the OT distance between G k and F , we reformulate the prediction probability as: [Equation]
[Equation]
In the outer loop, we fix the transport plan T ∗ and apply the cross entropy loss to optimize the{ vec l,n | L,Nl =1 ,n =1 } as:
[Equation]
where y x is a one-hot label vector.
The detail algorithm can be found in the supplementary materials.
Though the optimization strategy of the optimal transport and prompts is two-stage, the wholetraining flow is end-to-end.
It is because that the transport plan is computed using a small numberof matrix multiplications as one forward module of the neural network.
The gradients of thesematrix multiplications are taped for backpropagation for end-to-end optimization, which makes thewhole system fully differentiable (including the iterative algorithm) and easy to implement using anautograd library like PyTorch.
In the experiments, we found that it is natural and relatively easy tothis optimization strategy. 3.4 Inference strategy In the inference, given one query image and the learned prompts, we first obtain the a visual featureset containing M = H × W vectors and a prompt feature set containing N × C vectors. Then, wecalculate the distance between the visual feature set and the prompt feature set of each class by OTas (6) .
After obtaining the OT distance for each class, we sort the distance and classify the image.
Extensive experiments are conducted to evaluate our method, including comparison with CoOp,ablation studies, parameter analysis extensibility analysis, computing cost analysis and visualization.
4.1 Datasets
We followed the experimental settings in the CoOp [63] for the few-shot learning evaluation.
Theexperiments are conducted on the 11 visual recognition datasets, including Caltech101 [9], DTD [5],
EuroSAT [12], FGVCAircraft [29], Flowers102 [32], Food101 [3], ImageNet [7], OxfordPets [33],
StanfordCars [21], SUN397 [55], and UCF101 [47].
These datasets span visual classification ofgeneric objects, scenes, actions, fine-grained categories, and so on, which constitutes a comprehensiveevaluation of our method.
All experiments adopted the few-shot evaluation protocol used in CLIP [39]and CoOp [63], where we respectively choose 1, 2, 4, 8, and 16 shots for model training and use theoriginal test set for evaluation.
Besides, we also evaluated the robustness of our method with domainshift.
Following CoOp, we used the ImageNet as the source domain and evaluate our method with
ImageNet-based robustness evaluation datasets including ImageNetV2 [43], ImageNet-Sketch [52],
ImageNet-A [14], and ImageNet-R [13].
A detailed introduction of each dataset can be found in thesupplementary materials.
4.2 Implementation details
We chose CoOp [63] as our main competitor to evaluate our method.
Compared with CoOp whichonly learns a global prompt for one class, our PLOT method learns multiple local prompts and appliesthe OT distance for better fine-grained alignment.
Besides, we also reported the performance oftraining a linear classifier with the CLIP [39] features.
It is also a widely-used strategy to adapt thepretrained knowledge for the downstream task [46].
We reproduced the performance of CoOp andthe CLIP linear probe with the released official code.
The original CoOp method has different versions with different class token positions and parameterinitialization strategies.
We applied the default model that fixes the class token positions in the end dueto the limited performance gap between two different ways of positioning the class token.
Besides, weused the random parameter initialization strategy but not the class-specific context version.
Followingthe widely used setting in [63, 64, 10, 58], we also chose RN50 [11] as the backbone network of thevisual branch and set the length of learnable context tokens as 16.
All the code of our method is basedon CoOp, which adopted the SGD optimizer with 0.002 initial learning rate, CosineAnnealingLR [Table] schedule, and a warmup trick with 1e-5 learning rate.
Besides, we also followed the epoch strategy totrain more epochs for more shots.
We apply N = 4 prompts for each category and use M = 7 × 7 due to the feature map size.
We setthe hyper-parameters in the Sinkhorn distances algorithm
[6] as λ = 0 . 1 for all the datasets.
We setthe maximum iteration number of the inner loop as 100 and will early stop the iteration when theaverage absolute update value Λ < 0 .
We initialize all values in the vector v and µ as 1 /N and1 /M respectively.
All models are conducted on the Pytorch [34] 1.7.1 and trained on 4 NVIDIA A100 GPUs.
We repeated the experiments three times with different seeds and reported the average.
4.3 Comparison With CoOp
In this subsection, we compare our PLOT with the baseline CoOp on the few-shot recognition anddomain generalization tasks.
Few-Shot Learning We summarized the experimental results in Figure 3 where the red line denotesour PLOT method, the blue one denotes CoOp, the purple line denotes CoCoOp, and the green oneis the CLIP linear probe.
The detailed accuracy can be found in the supplementary materials.
Weobserved that both prompt learning methods (PLOT and CoOp) outperform the linear probe methodby a large margin.
Besides, PLOT can further improve the performance of CoOp and CoCoOp onmost of the datasets.
Taking the average accuracy (at the left top) as the example, Plot respectivelygained 3 .
61% performance boost over CoOp at 1 , 2 , 4 , 8 , 16 shots.
Wefound the performance gap will reduce when shots increase.
It is not surprising since both CoOpand PLOT focus on utilizing the pre-trained knowledge, and the effect of pre-training diminishesgiven more training data.
Among all datasets, PLOT achieves a larger improvement over CoOp onthe FOOD101 and DTD datasets and achieves comparable performance only on the StanfordCarsdatasets.
For the FGVCAircraft dataset in which the CoOp only obtains 7 .
77% accuracy, our PLOTcan achieve an accuracy of 17 .
79% , twice as high as that of the CoOp.
Note that we don’t use theclass-specific context, thus the performance on the fine-grained classification datasets is lower, e.g. [Table] FOOD [Table]
[Table]
the performance of both CoOp and PLOT without class-specific context is lower than the linearprobing on FGVCAircraft.
All these performance comparisons can serve as experimental evidence todemonstrate that multiple local prompts and optimal transport distance facilitate the prompt learningof vision-language models.
On StanfordCar, learning multiple prompts didn’t significantly improvethe performance over a single prompt. It may be because the discriminative characters in this datasetcoincide with each other, such that one global prompt and one global visual feature can work well.
Domain generalization The robustness also plays a critical role in model applications since thereal-world environment may have large domain shifts with the training data.
Therefore, we conducteda robustness evaluation to investigate the transferability of models learned by PLOT.
Table 1 summarizes the results of our PLOT method and CoOp on four ImageNet-based robustnessevaluation datasets.
For both methods, we trained the models on ImageNet with 16 shots per class.
For PLOT, we set the number of prompts as N = 4 .
We can observe that PLOT outperforms CoOpconsistently on both source and target domains.
These experimental results demonstrate that theperformance improvement of our learning multiple prompts doesn’t rely on single-domain overfitting.
4.4 Ablation Studies and More Analysis
In this subsection, we conducted the ablation studies to investigate the effectiveness of differentcomponents, in order to answer the following questions.
Q: Can we directly learn multiple prompts by respectively matching each prompt with theglobal visual feature?
A: No .
As shown in Table 2, we report the performance of directly matchingthe global visual feature (notated as “G”) and compare it with the baseline CoOp and our PLOT onthree datasets including Caltech101, DTD, and FOOD101.
We observe that there is no improvementover the baseline on some datasets (such as Caltech101 and FOOD101) if we only directly matchprompts and global features.
Though “G” obtained the improvement on the DTD dataset, thisimprovement is still less than that of PLOT.
It is because this “G” method is incentivized to learn theindistinguishable prompts, which contradicts our purpose to learn multiple comprehensive prompts.
We further add some constraints to push away the prompt from each other.
For example, we add anobjective function to add the distance between every two prompts as a regularization term, whichis notated as “V”.
However, comparing “G” and “G+V”, we do not find significant and consistentimprovement when using variance loss.
Q: Does the improvement mainly come from using all feature maps?
A: No .
In PLOT, we applyall feature maps of the visual encoder branch, where each feature is a local embedding at one spatialposition.
Compared with the global feature, these local features are more informative and containfine-grained clues.
However, we demonstrate that the improvement of PLOT does not only rely onusing all feature maps.
On the contrary, directly using the feature map to replace the global featurecauses a large performance drop.
For example, on all three datasets, directly using the feature map(“M” or “M+V”) has an around 20% 1 shot accuracy drop over using the global visual feature.
Itis not surprising since the original CLIP model is trained by matching the global visual feature andlanguage feature.
Without using the OT method, the distance between the feature map and multipletextual prompts degenerates to the mean distance of each feature-prompt pair.
Besides, when usingthe feature map, adding the variance loss works well, especially for more shots.
For example, theaccuracy on 16 shots DTD is improved by a large margin (from 26.99 to 56.85).
Q: How many prompts are needed?
A: 4 prompts are enough One important hyper-parameterin PLOT is the number of prompts.
To analyze the effect of the number of prompts, we conductedthe experiments on three datasets with 1 , 2 , 4 , 8 prompts.
The results are summarized in the whitepart of Table 3.
We can observe that the performance obviously increases when adding the numberof prompts from 1 to 4.
For example, PLOT (N=4) respectively obtains 1 .
64% , and 1 .
68%1-shot accuracy improvement over PLOT (N=1) on three datasets.
Besides, when we further increasethe number of prompts, the improvement is not consistent.
To balance the improvement and cost,we set N = 4 as the default configuration of our PLOT model.
In the experiments, we tuned thishyper-parameter on the Caltech101 dataset and applied it to other datasets.
Q: Can PLOT benefit zero-shot learning?
A: No . CLIP [39] shows that manually designing theprompts can still achieve good performance.
We obtain 7 prompts by prompt engineering on the
ImageNet dataset and can further ensemble them to obtain 60 .
38 % top 1 accuracy.
In this section,we replace the cosine distance between the global visual feature and prompt ensemble with the OT distance between the feature map and all 7 prompts.
However, without any learning, the OTdistance only obtains 58 .
78 % accuracy.
It is a limitation of the PLOT to still need few-shot datafor optimization, which cannot be directly applied in the zero-shot setting.
We argue there are tworeasons why the OT distance does not work without learning: 1) prompt engineering selects promptsbased on the global feature and cosine distance, instead of OT distance with feature map; 2) all theseselected prompts are closed to the global feature and lack the complementarity.
Q: Can PLOT benefit Adapter-based methods?
A:
Yes .
Adapter-based methods [10, 58] is anotherresearch direction of the efficient adaptation of pre-trained vision-language models.
Different fromthe prompt learning that fixes the model parameters and tunes the language prompt, adapter-basedmethods [10, 58] allow for fine-tuning a part of the network or adding an extra model for training.
Recently, adapter-based methods also achieve good performance on few-shot visual recognition.
Therefore, we want to explore whether our PLOT method can benefit them, and how.
We apply the Tip-adapter-F [58] as our baseline method, which learns a Linear ( d, N cls × K shots )model to describe one image by the similarity with all training samples, where d is the dimension ofvisual feature, N cls is the number of categories (e.g. 1000 in ImageNet), and K shots is the numberof shots.
Then, the final similarity consists of the original distance between the visual feature andprompt ensembling and the new distance calculated by the learned feature and one-hot vector oflabels (whose dimension is ( N cls × K shots , N cls ) ).
Please find details in Tip-adapter-F [58].
Tointroduce PLOT to this framework, we first used the feature map to replace the global feature and [Figure] [Table] Rooster
[Table] then learned multiple linear models. As a result, with different local features and different linearmodels, we can obtain a M × N distance matrix and apply the Sinkhorn algorithm [6] to calculatethe OT distance.
Furthermore, we can apply the learned prompts as co-partner of the ensemblingprompt to refine the final similarity.
Table 4 summarizes the few-shot recognition results of the original Tip-Adapter-F method and ouradapter-based PLOT methods on ImageNet.
From this table, We observe that using the OT distancecan improve the performance of the adapter-based method.
Using the learned prompts, we can furtherpromote the accuracy of all settings.
Q: What is the extra computation time cost of PLOT over CoOp baseline?
A: Around 10 %inference speed and 5 % training time .
Despite the performance improvement, the extra computationcost is still a limitation of PLOT.
Please see the detailed analysis in the supplementary materials.
4.5 Visualization
In this subsection, we provide some visualization examples of the transport plans T related to differentprompts (N=4).
We translate each transport plan into colorful heatmaps and resize them into theiroriginal size and combine them with the raw image.
As shown in Figure 4, we provide the heatmapsof 4 categories in ImageNet.
We observe that different transport plans highlight different regions ofthe image, which demonstrates that the learned multiple prompts are complementary.
For the class“Brambling”, the prompts respectively focus on the head, tail, wing, and environment.
For “Dog
Sled”, the prompts are related to dogs, the sled, some ties, and the snow environment.
Conclusion
In this paper, we present a method, named PLOT, to learn multiple comprehensive prompts todescribe diverse characteristics of one category.
To avoid convergence to one point, we propose toapply the optimal transport to achieve the fine-grained alignment between both vision and languagedomains.
We apply a two-stage optimization strategy where the inner loop fixes the prompts andlearns the transport plan to calculate the cross-modality distance, and the outer loop uses this distanceto optimize the prompt learner.
We build our method on the base of CoOp and achieve significantimprovement on the few-shot recognition task in various datasets, which demonstrates the advantageto learn multiple prompts instead of a single one.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu-
tions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See our analysis in Section 4.4. 1)
Ourmethod is still need few-shot data for optimization, which cannot be applied in zero-shot setting.
The method needs more computing cost than CoOp.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
We propose
a general framework for using the vision-language pre-trained model.
It is not for specific
applications, which does not directly involve societal issues.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments
[Table]
multiple times)?
[Yes]
[Table] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
[Yes] See Section 4.2.4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
b) Did you mention the license of the assets?
[N/A]
[Table]
) Did you include any new assets either in the supplemental material or as a URL?
[No]
[Table]
Did you discuss whether and how consent was obtained from people whose data you’re us-
[Table]
ing/curating?
[N/A]
[Table]
Did you discuss whether the data you are using/curating contains personally identifiable informa-
[Table]
tion or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
Did you include the text of instructions given to participants and screenshots,
applicable?
[Table]
you describe any potential participant risks, links to Institutional Review Board (
[Table]
approvals, if applicable?
[N/A] [Table]
Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation?
[N/A] [Table]