Original: V ARIATIONAL PROMPT TUNING IMPROVES GENERALIZATION OF VISION - LANGUAGE MODELS
Revised: V ARIATIONAL PROMPT TUNING IMPROVES GENERALIZATION OF VISION - LANGUAGE MODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Prompt tuning provides an efficient mechanism to adapt large vision-language models to downstream tasks by treating part of the input language prompts as learnable parameters while freezing the rest of the model.
Revised: Prompt tuning provides an efficient mechanism to adapt large vision-language models to downstream tasks by treating part of the input language prompts as learnable parameters while freezing the rest of the model.

Original: Existing works for prompt tuning are however prone to damaging the generalization capabilities of the foundation models, because the learned prompts lack the capacity of covering certain concepts within the language model.
Revised: Existing works for prompt tuning are however prone to damaging the generalization capabilities of the foundation models, because the learned prompts lack the capacity of covering certain concepts within the language model.

Original: To avoid such limitation, we propose a probabilistic modeling of the underlying distribution of prompts, allowing prompts within the support of an associated concept to be derived through stochastic sampling.
Revised: To avoid such limitation, we propose a probabilistic modeling of the underlying distribution of prompts, allowing prompts within the support of an associated concept to be derived through stochastic sampling.

Original: This results in a more complete and richer transfer of the information captured by the language model, providing better generalization capabilities for downstream tasks.
Revised: This results in a more complete and richer transfer of the information captured by the language model, providing better generalization capabilities for downstream tasks.

Original: The resulting algorithm relies on a simple yet powerful variational framework that can be directly integrated with other developments.
Revised: The resulting algorithm relies on a simple yet powerful variational framework that can be directly integrated with other developments.

Original: We show our approach is seamlessly integrated into both standard and conditional prompt learning frameworks, improving the performance on both cases considerably, especially with regards to preserving the generalization capability of the original model.
Revised: We show our approach is seamlessly integrated into both standard and conditional prompt learning frameworks, improving the performance on both cases considerably, especially with regards to preserving the generalization capability of the original model.

Original: Our method provides the current state-of-the-art for prompt learning, surpassing CoCoOp by 1.6% average Top-1 accuracy on the standard benchmark.
Revised: Our method provides the current state-of-the-art for prompt learning, surpassing CoCoOp by 1.6% average Top-1 accuracy on the standard benchmark.

Original: Remarkably, it even surpasses the original CLIP model in terms of generalization to new classes.
Revised: Remarkably, it even surpasses the original CLIP model in terms of generalization to new classes.

Original: Implementation code will be released.
Revised: Implementation code will be released.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In a continuous quest for better pre-training strategies, models based on image and language supervision have set impressive milestones, with CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021) and Flamingo (Alayrac et al., 2022) being leading examples.
Revised: In a continuous quest for better pre-training strategies, models based on image and language supervision have set impressive milestones, with CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021) and Flamingo (Alayrac et al., 2022) being leading examples.

Original: Contrastively trained vision-language models consist of image and text encoders that align semantically-related concepts in a joint embedding space.
Revised: Contrastively trained vision-language models consist of image and text encoders that align semantically-related concepts in a joint embedding space.

Original: Such models offer impressive zero-shot image classification by using the text encoder to generate classifier weights from arbitrarily newly defined category classes without relying on any visual data.
Revised: Such models offer impressive zero-shot image classification by using the text encoder to generate classifier weights from arbitrarily newly defined category classes without relying on any visual data.

Original: In particular, the class name is used within a handcrafted prompt template and then tokenized and encoded into the shared embedding space to generate new classifier weights.
Revised: In particular, the class name is used within a handcrafted prompt template and then tokenized and encoded into the shared embedding space to generate new classifier weights.

Original: Rather than manually defining prompts, Zhou et al.
Revised: Rather than manually defining prompts, Zhou et al.

Original: (2022b) and Lester et al.
Revised: (2022b) and Lester et al.

Original: (2021) proposed that prompts can be instead optimized in a data-driven manner through back-propagation by maximizing a cross-entropy loss on the downstream task.
Revised: (2021) proposed that prompts can be instead optimized in a data-driven manner through back-propagation by maximizing a cross-entropy loss on the downstream task.

Original: However, despite the performance improvement on downstream tasks, prompt learning negatively affects the generalization capability of the vision-language model.
Revised: However, despite the performance improvement on downstream tasks, prompt learning negatively affects the generalization capability of the vision-language model.

Original: While subsequent works have focused on how to bridge the generalization gap, e.g. Zhou et al.
Revised: While subsequent works have focused on how to bridge the generalization gap, e.g. Zhou et al.

Original: (2022a); Zhu et al.
Revised: (2022a); Zhu et al.

Original: (2022), in practice the generalization power of the foundation model is significantly degraded.
Revised: (2022), in practice the generalization power of the foundation model is significantly degraded.

Original: Our work tackles this same problem, as it seeks to improve downstream performance but without degrading the generalization capability of the original model.
Revised: Our work tackles this same problem, as it seeks to improve downstream performance but without degrading the generalization capability of the original model.

Original: To do so, we propose a data-driven method for directly learning the underlying distribution within the prompt space associated to the target concept.
Revised: To do so, we propose a data-driven method for directly learning the underlying distribution within the prompt space associated to the target concept.

Original: In particular, we frame prompt tuning as a variational inference problem, where a base learned prompt is combined with a residual vector sampled from the instance-specific underlying distribution.
Revised: In particular, we frame prompt tuning as a variational inference problem, where a base learned prompt is combined with a residual vector sampled from the instance-specific underlying distribution.

Original: This formulation provides two advantages.
Revised: This formulation provides two advantages.

Original: First, it investigates the prompt space more thoroughly and results in more informative use of the language space, leading to better generalization.
Revised: First, it investigates the prompt space more thoroughly and results in more informative use of the language space, leading to better generalization.

Original: Second, it enables us to boost performance by capturing the uncertainty information in fine-grained classification problems.
Revised: Second, it enables us to boost performance by capturing the uncertainty information in fine-grained classification problems.

Original: The resulting approach is orthogonal to standard prompt learning approaches, being effective when combined with both standard (Zhou et al., 2022b) and conditional (Zhou et al., 2022a) approaches.
Revised: The resulting approach is orthogonal to standard prompt learning approaches, being effective when combined with both standard (Zhou et al., 2022b) and conditional (Zhou et al., 2022a) approaches.

Original: In fact, when combined with the conditional approach, our method maintains the gains on the seen classes provided by the conditional method while simultaneously matching or even surpassing the generalization capability on unseen classes of the original vision-language model.
Revised: In fact, when combined with the conditional approach, our method maintains the gains on the seen classes provided by the conditional method while simultaneously matching or even surpassing the generalization capability on unseen classes of the original vision-language model.

Original: In summary, our contributions in this paper are as follows:
Revised: In summary, our contributions in this paper are as follows:

Original: We propose a variational framework that is capable of capturing the general or instance specific distribution within the prompt space.
Revised: We propose a variational framework that is capable of capturing the general or instance specific distribution within the prompt space.

Original: Since generalization is obtained through transfer from the language space, we obtain better generalization capability.
Revised: Since generalization is obtained through transfer from the language space, we obtain better generalization capability.

Original: We show that the proposed approach is orthogonal to recent developments, and can be successfully combined with both standard and conditional prompt learning variants.
Revised: We show that the proposed approach is orthogonal to recent developments, and can be successfully combined with both standard and conditional prompt learning variants.

Original: We empirically show that our proposed method improves performance and provides better generalization, leading to state-of-the-art accuracy in 24 out of 28 standard benchmarks set forth by prior work, surpassing CoCoOp by 1.6% average Top-1 accuracy.
Revised: We empirically show that our proposed method improves performance and provides better generalization, leading to state-of-the-art accuracy in 24 out of 28 standard benchmarks set forth by prior work, surpassing CoCoOp by 1.6% average Top-1 accuracy.

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Prompt learning in NLP.
Revised: Prompt learning in NLP.

Original: Prompt learning was originally proposed within the NLP domain, following the appearance of foundation models such as GPT-3 (Brown et al., 2020).
Revised: Prompt learning was originally proposed within the NLP domain, following the appearance of foundation models such as GPT-3 (Brown et al., 2020).

Original: Early prompt learning methods constructed prompts by combining words in the language space such that the model would perform better on downstream evaluation (Shin et al., 2020; Jiang et al., 2020).
Revised: Early prompt learning methods constructed prompts by combining words in the language space such that the model would perform better on downstream evaluation (Shin et al., 2020; Jiang et al., 2020).

Original: Subsequent methods, e.g. Li & Liang (2021); Lester et al.
Revised: Subsequent methods, e.g. Li & Liang (2021); Lester et al.

Original: (2021), prepend a set of learnable prompts to the input of a frozen model and optimize through back-propagation, which allows better flexibility than using existing words, at the cost of leading to prompts that do not correspond to an actual phrase.
Revised: (2021), prepend a set of learnable prompts to the input of a frozen model and optimize through back-propagation, which allows better flexibility than using existing words, at the cost of leading to prompts that do not correspond to an actual phrase.

Original: Instead, He et al.
Revised: Instead, He et al.

Original: (2022) focus on a multi-task scenario and use a HyperNetwork to conditionally generate task-specific and layer-specific prompts that are pre-pended to the values and keys inside the self-attention layers of a the frozen model.
Revised: (2022) focus on a multi-task scenario and use a HyperNetwork to conditionally generate task-specific and layer-specific prompts that are pre-pended to the values and keys inside the self-attention layers of a the frozen model.

Original: Within the NLP domain, prompt learning has also been shown to work better than in-context learning (Liu et al., 2022).
Revised: Within the NLP domain, prompt learning has also been shown to work better than in-context learning (Liu et al., 2022).

Original: Prompting in Vision and Language models.
Revised: Prompting in Vision and Language models.

Original: Research on prompt learning for vision-language models have been largely inspired by prior work within NLP.
Revised: Research on prompt learning for vision-language models have been largely inspired by prior work within NLP.

Original: Similar to e.g. Li & Liang (2021),
Revised: Similar to e.g. Li & Liang (2021),

Original: CoOp (Zhou et al., 2022b) proposes a prompt learning method that optimizes unified or class specific prompts in the continuous space through back-propagation.
Revised: CoOp (Zhou et al., 2022b) proposes a prompt learning method that optimizes unified or class specific prompts in the continuous space through back-propagation.

Original: While CoOp obtains good accuracy on downstream tasks, it negatively affects the generalization ability to new unseen classes.
Revised: While CoOp obtains good accuracy on downstream tasks, it negatively affects the generalization ability to new unseen classes.

Original: CoCoOp (Zhou et al., 2022a) extends CoOp and partially bridges the generalization gap by generating instance-specific prompt residuals through a conditioning mechanism dependent on the visual data.
Revised: CoCoOp (Zhou et al., 2022a) extends CoOp and partially bridges the generalization gap by generating instance-specific prompt residuals through a conditioning mechanism dependent on the visual data.

Original: ProGrad (Zhu et al., 2022) shares the same goal as CoCoOp of bridging the generalization gap, but instead proposes to match the gradient of the prompt to the general knowledge of the CLIP model to prevent prompt tuning from forgetting the general knowledge learned from the foundation model.
Revised: ProGrad (Zhu et al., 2022) shares the same goal as CoCoOp of bridging the generalization gap, but instead proposes to match the gradient of the prompt to the general knowledge of the CLIP model to prevent prompt tuning from forgetting the general knowledge learned from the foundation model.

Original: Alternative directions consist of test-time prompt tuning (Shu et al., 2022), where consistency across multiple views is used as the supervisory signal, and unsupervised prompt learning (Huang et al., 2022), where a pseudo-labelling strategy is proposed instead to obtain the labels needed to drive the prompt learning.
Revised: Alternative directions consist of test-time prompt tuning (Shu et al., 2022), where consistency across multiple views is used as the supervisory signal, and unsupervised prompt learning (Huang et al., 2022), where a pseudo-labelling strategy is proposed instead to obtain the labels needed to drive the prompt learning.

Original: Perhaps the most similar work to ours is Lu et al.
Revised: Perhaps the most similar work to ours is Lu et al.

Original: In this work, the authors use an ensemble of prompts and model their distribution within the language embedding space, with optimization seeking to minimize the negative log-likelihood with respect to the corresponding visual embedding.
Revised: In this work, the authors use an ensemble of prompts and model their distribution within the language embedding space, with optimization seeking to minimize the negative log-likelihood with respect to the corresponding visual embedding.

Original: Unlike ours, their method relies on hand-crafted rules to define the prompt ensemble, thus still relying on the effectiveness of hand-crafted designs.
Revised: Unlike ours, their method relies on hand-crafted rules to define the prompt ensemble, thus still relying on the effectiveness of hand-crafted designs.

Original: The number of learnable prompts is also pre-defined, potentially offering sub-optimal coverage of an NLP concept.
Revised: The number of learnable prompts is also pre-defined, potentially offering sub-optimal coverage of an NLP concept.

Original: Finally, it is not clear how to apply their strategy within the context of conditional prompt learning.
Revised: Finally, it is not clear how to apply their strategy within the context of conditional prompt learning.

Original: We believe that modelling the input prompt space rather than relying on a fixed number of templates is a more powerful and flexible approach.
Revised: We believe that modelling the input prompt space rather than relying on a fixed number of templates is a more powerful and flexible approach.

Original: We provide empirical evidence of the superiority of our approach in the experiments.
Revised: We provide empirical evidence of the superiority of our approach in the experiments.

Original: While beyond our current scope, it is worth noting that prompt learning has been applied to a wider range of problems and scenarios, which highlights its power and flexibility.
Revised: While beyond our current scope, it is worth noting that prompt learning has been applied to a wider range of problems and scenarios, which highlights its power and flexibility.

Original: Among them are important topics such as unsupervised domain adaptation (Ge et al., 2022), multi-label classification (Sun et al., 2022), video classification (Ju et al., 2022), object detection (Du et al., 2022; Feng et al., 2022) 
Revised: Among them are important topics such as unsupervised domain adaptation (Ge et al., 2022), multi-label classification (Sun et al., 2022), video classification (Ju et al., 2022), object detection (Du et al., 2022; Feng et al., 2022) or pixel-level labelling (Rao et al., 2022).

Revised: Finally, prompt learning as a means to adapt pre-trained models has also been applied to purely vision models (Jia et al., 2022; Sandler et al., 2022) providing great parameter efficiency.

Original:  M ETHOD 3. B ACKGROUND
Revised:  M ETHOD 3. B ACKGROUND

Original: Contrastive Language-Image Pretraining (CLIP) (Radford et al., 2021) consists of an image encoder f
Revised: Contrastive Language-Image Pretraining (CLIP) (Radford et al., 2021) consists of an image encoder f

Original: ( x ) and text encoder g ( t ) , each producing a d -dimensional ( L 2 normalized) embedding from an arbitrary image x ∈ R 3 × H × W , and word embeddings t ∈ R L × e , with L representing the text length and e the embedding dimension 1 .
Revised: ( x ) and text encoder g ( t ) , each producing a d -dimensional ( L 2 normalized) embedding from an arbitrary image x ∈ R 3 × H × W , and word embeddings t ∈ R L × e , with L representing the text length and e the embedding dimension 1 .

Original: Both encoders are trained together using a contrastive loss from a large-scale dataset composed of paired images and captions.
Revised: Both encoders are trained together using a contrastive loss from a large-scale dataset composed of paired images and captions.

Original: Once trained,
Revised: Once trained,

Original: CLIP can be used for zero-shot C -class image classification by generating each of the c classifier weights w c as the d -dimensional text encoding g ( t c ) .
Revised: CLIP can be used for zero-shot C -class image classification by generating each of the c classifier weights w c as the d -dimensional text encoding g ( t c ) .

Original: Here t c results from adding the classspecific word embedding e c to a pre-defined prompt p ∈ R L − 1 × e , i.e., w c = g ( t c ) with t c
Revised: Here t c results from adding the classspecific word embedding e c to a pre-defined prompt p ∈ R L − 1 × e , i.e., w c = g ( t c ) with t c

Original: = { p , e c } .
Revised: = { p , e c } .

Original: The prompt p is manually crafted to capture the semantic meaning of the downstream task, e.g., t c = “ An image of a { class } ” .
Revised: The prompt p is manually crafted to capture the semantic meaning of the downstream task, e.g., t c = “ An image of a { class } ” .

Original: The probability of image x being classified as y ∈ { 1 ...C } is thus defined as p ( y | x 
Revised: The probability of image x being classified as y ∈ { 1 ...C } is thus defined as p ( y | x [Equation] =

Original: [Equation]
Revised: [Equation]

Original: Context Optimization (CoOp) (Zhou et al., 2022b) provides a learned alternative to manually defining prompts.
Revised: Context Optimization (CoOp) (Zhou et al., 2022b) provides a learned alternative to manually defining prompts.

Original: CoOp learns a fixed prompt from a few annotated samples.
Revised: CoOp learns a fixed prompt from a few annotated samples.

Original: The prompt is designed as a learnable embedding matrix p ∈ R L × e which is updated via back-propagating the classification error through the frozen CLIP model.
Revised: The prompt is designed as a learnable embedding matrix p ∈ R L × e which is updated via back-propagating the classification error through the frozen CLIP model.

Original: Specifically, for a set of N annotated meta-training samples { x i , y i } Ni =1 , the prompt p is obtained by minimizing the cross-entropy loss, as:[Equation]
Revised: Specifically, for a set of N annotated meta-training samples { x i , y i } Ni =1 , the prompt p is obtained by minimizing the cross-entropy loss, as:

Original: [Equation]
Revised: [Equation]

Original: Note that this approach, while resembling that of common meta-learning approaches, can still be deployed in a zero-shot scenario provided that for new classes the classification weights will be given by the text encoder.
Revised: Note that this approach, while resembling that of common meta-learning approaches, can still be deployed in a zero-shot scenario provided that for new classes the classification weights will be given by the text encoder.

Original: Although this approach allows to generalize to new tasks with few training iterations, learning a fixed prompt is sensitive to domain shifts between the annotated samples and the test set.
Revised: Although this approach generalizes to new tasks with few training iterations, learning a fixed prompt is sensitive to domain shifts between the annotated samples and the test set.

Original: Conditional Prompt Learning (CoCoOp) (Zhou et al., 2022a) attempts to overcome domain shifts by learning an instance-specific continuous prompt that is conditioned on the input image.
Revised: Conditional Prompt Learning (CoCoOp) (Zhou et al., 2022a) attempts to overcome domain shifts by learning an instance-specific continuous prompt that is conditioned on the input image.

Original: To ease the training of a conditional prompt generator, CoCoOp defines each conditional token in a residual way, with a task-specific, learnable set of tokens p and a residual vector that is conditioned on the input image.
Revised: To ease the training of a conditional prompt generator, CoCoOp defines each conditional token in a residual way, with a task-specific, learnable set of tokens p and a residual vector that is conditioned on the input image.

Original: Assuming p to be composed of L learnable tokens p =[
Revised: Assuming p to be composed of L learnable tokens p =[

Original: p 1 , p 2 , ..., p L ] , the residual vector r ( x )=
Revised: p 1 , p 2 , ..., p L ] , the residual vector r ( x )=

Original: π ϕ ( f ( x ))
Revised: π ϕ ( f ( x ))

Original: ∈
Revised: ∈

Original: R e is produced by a small neural network π ϕ with as input the image features f ( x ) .
Revised: R e is produced by a small neural network π ϕ with as input the image features f ( x ) .

Original: The new prompt is then computed as p ( x )=[ p 1 + r ( x ) , p 2 + r ( x ) , ..., p L + r ( x )] .
Revised: The new prompt is then computed as p ( x )=[ p 1 + r ( x ) , p 2 + r ( x ) , ..., p L + r ( x )] .

Original: The training now comprises learning the task-specific prompt p and the parameters ϕ of the neural network π ϕ .
Revised: The training now comprises learning the task-specific prompt p and the parameters ϕ of the neural network π ϕ .

Original: Defining the context-specific text embedding t c ( x )= { p ( x ) , e c } , and p ( y | x ) as :
Revised: Defining the context-specific text embedding t c ( x )= { p ( x ) , e c } , and p ( y | x ) as :

Original: [Equation]
Revised: [Equation]

Original: (2) the learning is formulated as:
Revised: the learning is formulated as:

Original: [Equation]
Revised: [Equation]

Original: While CoCoOp achieves state-of-the-art results in a large variety of downstream tasks, it is still prone to the domain shift problem, considering that π ϕ provides a deterministic residual vector from the image features f ( x ) which are expected to be domain-specific.
Revised: While CoCoOp achieves state-of-the-art results in a large variety of downstream tasks, it is still prone to the domain shift problem, considering that π ϕ provides a deterministic residual vector from the image features f ( x ) which are expected to be domain-specific.

Original: Prompt Distribution Learning (ProDA) (Lu et al., 2022) is work concurrent to CoCoOp that focuses learning a distribution of prompts that generalize to a broader set of tasks.
Revised: Prompt Distribution Learning (ProDA) (Lu et al., 2022) is work concurrent to CoCoOp that focuses learning a distribution of prompts that generalize to a broader set of tasks.

Original: ProDA proposes to learn a collection of prompts P = { p k } K k =1 that can be used to subsequently generate an a posteriori distribution of the classifier weights for each of the target classes.
Revised: ProDA proposes to learn a collection of prompts P = { p k } K k =1 that can be used to subsequently generate an a posteriori distribution of the classifier weights for each of the target classes.

Original: For a given mini-batch of K sampled prompts p k ∼ P , the classifier weights w c are sampled from the posterior distribution N ( µ w 1: C , Σ w 1: C ) , with mean µ w 1: C and covariance Σ w 1: C computed from the collection { w k,c
Revised: For a given mini-batch of K sampled prompts p k ∼ P , the classifier weights w c are sampled from the posterior distribution N ( µ w 1: C , Σ w 1: C ) , with mean µ w 1: C and covariance Σ w 1: C computed from the collection { w k,c

Original: = g ( t k,c ) } c =1: C,k =1: K , with t k,c
Revised: = g ( t k,c ) } c =1: C,k =1: K , with t k,c

Original: = { p k , e c } .
Revised: = { p k , e c } .

Original: The objective is now formulated as:
Revised: The objective is now formulated as:

Original: [Equation]
Revised: [Equation]

Original: Computing E w l p ( y i | x i , w l )] is intractable and an upper bound to Eq.
Revised: Computing E w l p ( y i | x i , w l )] is intractable and an upper bound to Eq.

Original: 4 is derived.
Revised: 4 is derived.

Original: During inference, the classifier weights are set to those given by the predictive mean w c = µ w 1: C , computed across the collection of learned prompts P .
Revised: During inference, the classifier weights are set to those given by the predictive mean w c = µ w 1: C , computed across the collection of learned prompts P .

Original: While showing promising results compared to CoOp, how to combine it with the conditional prompt learning framework of CoCoOp is unclear.
Revised: While showing promising results compared to CoOp, how to combine it with the conditional prompt learning framework of CoCoOp is unclear.

Original: 3. V ARIATIONAL P ROMPT T UNING
Revised: 3. V ARIATIONAL P ROMPT T UNING

Original: In this paper, we propose to model the input prompt space in a probabilistic manner, as an a priori, instance-specific distribution.
Revised: In this paper, we propose to model the input prompt space in a probabilistic manner, as an a priori, instance-specific distribution.

Original: In particular, we define a distribution p γ over the prompts p that is instance-specific, i.e. p ∼ p γ ( x ) .
Revised: In particular, we define a distribution p γ over the prompts p that is instance-specific, i.e. p ∼ p γ ( x ) .

Original: To this end, we assume that p can be split into a fixed set of prompts p i and an instance specific residual vector r that act as a latent variable over p .
Revised: To this end, we assume that p can be split into a fixed set of prompts p i and an instance specific residual vector r that act as a latent variable over p .

Original: The instance-specific prompt is then defined as:
Revised: The instance-specific prompt is then defined as:

Original: [Equation]
Revised: [Equation]

Original: where p γ ( x ) refers to the real posterior distribution over r conditioned on the observed features x .
Revised: where p γ ( x ) refers to the real posterior distribution over r conditioned on the observed features x .

Original: Denoting the class-specific input as t c,γ ( x )= { p γ ( x ) , e c } , the marginal likelihood p ( y | x ) is now defined as:
Revised: Denoting the class-specific input as t c,γ ( x )= { p γ ( x ) , e c } , the marginal likelihood p ( y | x ) is:

Original: [Equation]
Revised: [Equation]

Original: Solving Eq.
Revised: Solving Eq.

Original: 3 with the marginal likelihood defined as in Eq.
Revised: 3 with the marginal likelihood defined as in Eq.

Original: 6 is intractable, as it requires computing p γ ( r | x ) p γ ( x ) .
Revised: 6 is intractable, as it requires computing p γ ( r | x ) p γ ( x ) .

Original: Instead, we resort to deriving a lower bound, by introducing a variational posterior distribution π ϕ ( x ) from which the residual r γ can be sampled.
Revised: Instead, we resort to deriving a lower bound, by introducing a variational posterior distribution π ϕ ( x ) from which the residual r γ can be sampled.

Original: The variational bound is defined as:
Revised: The variational bound is defined as:

Original: [Equation]
Revised: [Equation]

Original: with p ( y | x , r ) ∝ e f ( x ) T g ( t c,γ ( x )) , where the dependency on r comes through the definition of t c,γ .
Revised: with p ( y | x , r ) ∝ e f ( x ) T g ( t c,γ ( x )) , where the dependency on r comes through the definition of t c,γ .

Original: The variational posterior distribution π ϕ plays a role akin to the metanet in CoCoOp.
Revised: The variational posterior distribution π ϕ plays a role akin to the metanet in CoCoOp.

Original: We thus refer to it as metanet in the following to align terminology.
Revised: We thus refer to it as metanet in the following to align terminology.

Original: Following standard variational optimization practices (Kingma & Welling, 2014; Gordon et al., 2019), we define π ϕ as a Gaussian distribution conditioned on the input image features x , as r ( x ) ∼ N ( µ ( x ) , Σ( x )) , with µ and Σ parameterized by two linear layers placed on top of the metanet π ϕ
Revised: Following standard variational optimization practices (Kingma & Welling, 2014; Gordon et al., 2019), we define π ϕ as a Gaussian distribution conditioned on the input image features x , as r ( x ) ∼ N ( µ ( x ) , Σ( x )) , with µ and Σ parameterized by two linear layers placed on top of the metanet π ϕ

Original: (see Figure 1).
Revised: (see Figure 1).

Original: The prior p γ ( r ) is defined as
Revised: The prior p γ ( r ) is defined as

Original: N ( 0 , I ) , and we make use of the reparameterization trick to generate Monte-Carlo samples from π ϕ to maximize the right side of Eq.
Revised: N ( 0 , I ) , and we make use of the reparameterization trick to generate Monte-Carlo samples from π ϕ to maximize the right side of Eq.

Original: The optimization of Eq.
Revised: The optimization of Eq.

Original: 7 comprises learning the prompt embeddings { p i } Li =1 as well as the parameters of the metanet π ϕ and the linear layers parameterizing µ and Σ .
Revised: 7 comprises learning the prompt embeddings { p i } Li =1 as well as the parameters of the metanet π ϕ and the linear layers parameterizing µ and Σ .

Original: Note that this adds little complexity as it requires learning p and π ϕ , given that µ and Σ are defined as two linear layers on top of π ϕ .
Revised: Note that this adds little complexity as it requires learning p and π ϕ , given that µ and Σ are defined as two linear layers on top of π ϕ .

Original: Inference.
Revised: Inference.

Original: At test time, K residuals are sampled from the conditional distribution π ϕ ( x ) , which are used to generate K different prompts per class p k =[ p 1 + r k , p 2 + r k , · · · , p L + r k ] .
Revised: At test time, K residuals are sampled from the conditional distribution π ϕ ( x ) , which are used to generate K different prompts per class p k =[ p 1 + r k , p 2 + r k , · · · , p L + r k ] .

Original: Each prompt is prepended to the class-specific embedding to generate a series of K separate classifier weights w k,c .
Revised: Each prompt is prepended to the class-specific embedding to generate a series of K separate classifier weights w k,c .

Original: We then compute p ( y = c | x )=(1 /K )
Revised: We then compute p ( y = c | x )=(1 /K )

Original: (cid:80)
Revised: (cid:80)

Original: Kk =1 p ( y = c
Revised: Kk =1 p ( y = c

Original: | x , w k,c ) and select ˆ c = arg max c p ( y = c
Revised: | x , w k,c ) and select ˆ c = arg max c p ( y = c

Original: | x ) as the predicted class.
Revised: | x ) as the predicted class.

Original: It is worth noting that because the posterior distribution is generated by the text encoder, it is not expected that for K → ∞ , (1 /K )
Revised: It is worth noting that because the posterior distribution is generated by the text encoder, it is not expected that for K → ∞ , (1 /K )

Original: (cid:80) k p ( y = c
Revised: (cid:80) k p ( y = c

Original: | x , w c,k ) → p ( y = c
Revised: | x , w c,k ) → p ( y = c

Original: | x , g ( { µ ( x ) , e c } ) , meaning that sampling at inference time remains relevant.
Revised: | x , g ( { µ ( x ) , e c } ) , meaning that sampling at inference time remains relevant.

Original: We study the dependency on the number of samples in the ablations.
Revised: We study the dependency on the number of samples in the ablations.

Original: Notably, our framework can be also used with the unconditional setting of CoOp, by simply removing the dependency of the input image from the latent distribution.
Revised: Notably, our framework can be also used with the unconditional setting of CoOp, by simply removing the dependency of the input image from the latent distribution.

Original: In such scenario, we keep a fixed set of prompt embeddings and learn a global latent distribution p γ over the residual vectors r .
Revised: In such scenario, we keep a fixed set of prompt embeddings and learn a global latent distribution p γ over the residual vectors r , as r ∼ N ( µ, Σ) , where µ and Σ are parameterized by two learnable vectors.

Original: In this case, p γ is a general distribution learned during training with no dependency on the input sample x .
Revised: In this case, p γ is a general distribution learned during training with no dependency on the input sample x .

Original: We show that CoOp with our proposed approach improves generalization on new classes.
Revised: We show that CoOp with our proposed approach improves generalization on new classes.

Original:  E XPERIMENTS AND R ESULTS 4. E XPERIMENTAL SETUP
Revised:  E XPERIMENTS AND R ESULTS 4. E XPERIMENTAL SETUP

Original: We follow the exact experimental setup of CoCoOp (Zhou et al., 2022a),which is currently the state-of-the-art approach for prompt tuning for vision-language models.
Revised: We follow the exact experimental setup of CoCoOp (Zhou et al., 2022a), currently the state-of-theart for prompt tuning for vision-language models.

Original: We describe the setup in the following for completeness.
Revised: We describe the setup in the following.

Original: Three tasks and fifteen datasets.
Revised: Three tasks and fifteen datasets.

Original: We evaluate variational prompt tuning for three different tasks: base-to-new generalization , cross-dataset transfer , and domain generalization .
Revised: We evaluate variational prompt tuning for three different tasks: base-to-new generalization , cross-dataset transfer , and cross-domain generalization .

Original: For base-to-new generalization and cross-dataset transfer tasks, we rely on the same 11 image recognition datasets as Zhou et al.
Revised: For baseto-new generalization and cross-dataset transfer tasks, we rely on the same 11 image recognition datasets as Zhou et al.

Original: (2022b;a).
Revised: (2022b;a).

Original: These include generic image classification datasets (ImageNet by (Deng et al., 2009) and Caltech101 by (Fei-Fei et al., 2004)), fine-grained classification datasets (OxfordPets by (Parkhi et al., 2012), StanfordCars by (Krause et al., 2013), Flowers102 by (Nilsback & Zisserman, 2008), Food101 by (Bossard et al., 2014) and FGVCAircraft by (Maji et al., 2013)), scene recognition (SUN397 by (Xiao et al., 2010)), action recognition (UCF101 by (Soomro et al., 2012)), texture classification (DTD by (Cimpoi et al., 2014)), and satellite imagery recognition (EuroSAT by (Helber et al., 2019)).
Revised: These include generic image classification datasets (ImageNet by (Deng et al., 2009) and Caltech101 by (Fei-Fei et al., 2004)), fine-grained classification datasets (OxfordPets by (Parkhi et al., 2012), StanfordCars by (Krause et al., 2013), Flowers102 by (Nilsback & Zisserman, 2008), Food101 by (Bossard et al., 2014) and FGVCAircraft by (Maji et al., 2013)), scene recognition (SUN397 by (Xiao et al., 2010)), action recognition (UCF101 by (Soomro et al., 2012)), texture classification (DTD by (Cimpoi et al., 2014)), and satellite imagery recognition (EuroSAT by (Helber et al., 2019)).

Original: For the domain generalization task, wefirst train our model on ImageNet and reportour evaluation metric on ImageNetV2 (Recht et al., 2019), ImageNet-Sketch (Wang et al., 2019),
Revised: For the cross-domain generalization task, we train our model on ImageNet and report on ImageNetV2 (Recht et al., 2019), ImageNet-Sketch (Wang et al., 2019),

Original: ImageNet-A (Hendrycks et al., 2021b), and ImageNet-R (Hendrycks et al., 2021a).
Revised: ImageNet-A (Hendrycks et al., 2021b), and ImageNet-R (Hendrycks et al., 2021a).

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We report average accuracy and harmonic mean H =2 × ( base × new ) /
Revised: We report average accuracy and harmonic mean H =2 × ( base × new ) /

Original: ( base + new ) (Xian et al., 2017) for the base-to-new generalization tasks.
Revised: ( base + new ) (Xian et al., 2017) for the base-to-new generalization tasks.

Original: For cross-dataset transfer learning and domain adaptation, we provide average accuracy results.
Revised: For cross-dataset transfer learning and domain adaptation, we provide average accuracy results.

Original: Baselines.
Revised: Baselines.

Original: We compare against zero-shot CLIP (Radford et al., 2021), CoOp (Zhou et al., 2022b),
Revised: We compare against zero-shot CLIP (Radford et al., 2021), CoOp (Zhou et al., 2022b),

Original: CoCoOp
Revised: CoCoOp

Original: (Zhou et al., 2022a), and ProDA (Lu et al., 2022).
Revised: (Zhou et al., 2022a), and ProDA (Lu et al., 2022).

Original: For zero-shot CLIP, CoOp, CoCoOp, all results are adopted from (Zhou et al., 2022a), and we reproduce all results for ProDA.
Revised: For zero-shot CLIP, CoOp, CoCoOp, all results are adopted from (Zhou et al., 2022a), and we reproduce all results for ProDA.

Original: Implementation details.
Revised: Implementation details.

Original: Our variational prompt tuning contains three sub-networks: an image encoder f ( x ) , a text encoder g ( t ) , and a metanet π ϕ .
Revised: Our variational prompt tuning contains three sub-networks: an image encoder f ( x ) , a text encoder g ( t ) , and a metanet π ϕ .

Original: The image encoder f ( x ) and text encoder g ( t ) are a ViT-B/16 (Dosovitskiy et al., 2021) and transformer (Vaswani et al., 2017), which are initialized with CLIP’s pre-trained weights and kept frozen during training, as in Zhou et al.
Revised: The image encoder f ( x ) and text encoder g ( t ) are a ViT-B/16 (Dosovitskiy et al., 2021) and transformer (Vaswani et al., 2017), which are initialized with CLIP’s pre-trained weights and kept frozen during training, as in Zhou et al.

Original: (2022b;a).
Revised: (2022b;a).

Original: The metanet π ϕ consists of two linear layers followed by ELU activation function as trunk and two linear heads on top to estimate the µ and Σ of the residual distribution.
Revised: The metanet π ϕ consists of two linear layers followed by ELU activation function as trunk and two linear heads on top to estimate the µ and Σ of the residual distribution.

Original: For each task and dataset, we optimize the number of samples K and epochs.
Revised: For each task and dataset, we optimize the number of samples K and epochs.

Original: Other hyper-parameters as well as the training Average ImageNet Caltech
Revised: Other hyper-parameters as well as the training pipeline in terms of few-shot task definitions are identical to Zhou et al.

Original: OxfordPets

Original: StanfordCars

Original: Flowers

Original: Food

Original: FGVCAircraft

Original: SUN

Original: DTD

Original: EuroSAT

Original: UCF +1.

Original: +0.

Original: +0.

Original: +2.

Original: +0.

Original: +5.

Original: +2.

Original: +3.

Original: +3.

Original: +1. +6. [Figure] +1.
Revised: (2022b;a) (see table 6 andin the appendix).

Original: +0. +0. +0. +1.
Revised: Implementation code will be released.

Original: +0.

Original: +6.

Original: +0.

Original: +2.

Original: +6.

Original: +1.

Original: (b) ∆( H ) =

Original: H (

Original: [Figure]
Revised: [Figure]

Original: +VPT ) − H ( CoCoOp ) pipeline in terms of few-shot task definitions are identical to Zhou et al.

Original: (2022b;a) (see table 6 andin the appendix).

Original: Implementation code will be released.

Original: 4. B ASE - TO - NEW GENERALIZATION
Revised: 4. B ASE - TO - NEW GENERALIZATION

Original: Setup.
Revised: Setup.

Original: We report the few-shot generalization of our method on 11 datasets for three different random seeds.
Revised: We report the few-shot generalization of our method on 11 datasets for three different random seeds.

Original: Each dataset is divided into two disjoint subsets: base classes and new classes.
Revised: Each dataset is divided into two disjoint subsets: base classes and new classes.

Original: We train our method on base classes and evaluate it on both base and new classes.
Revised: We train our method on base classes and evaluate it on both base and new classes.

Original: For a fair comparison, we follow Zhou et al.
Revised: For a fair comparison, we follow Zhou et al.

Original: (2022a;a) in terms of dataset split and number of shots (see Table 6 appendix.)
Revised: (2022a;a) in terms of dataset split and number of shots (see Table 6 appendix.)

Original: Results.
Revised: Results.

Original: From the results in Table 1, the lack of generalization capability of the CoOp approach is evidenced by the considerable discrepancy between the base and new classes’ accuracy.
Revised: From the results in Table 1, the lack of generalization capability of the CoOp approach is evidenced by the considerable discrepancy between the base and new classes’ accuracy.

Original: This is expected as CoOp only observes a small number of training samples to adapt the CLIP model for downstream tasks, resulting in overfitting.
Revised: This is expected as CoOp only observes a small number of training samples to adapt the CLIP model for downstream tasks, resulting in overfitting.

Original: Adding variational prompt tuning to CoOp reduces overfitting to base classes, improving the performance of new classes by 11 .
Revised: Adding variational prompt tuning to CoOp reduces overfitting to base classes, improving the performance of new classes by 11 .

Original: 54% and the harmonic mean by 1 . 7% across all datasets, at the expense of a base classes accuracy drop by 10 . 7% .
Revised: 54% and the harmonic mean by 1 . 7% across all datasets, at the expense of a base classes accuracy drop by 10 . 7% .

Original: Figure 2(a) depicts the relative improvement of our proposed strategy when compared to CoOp in terms of the harmonic mean, where we observe an improvement in 10 out of 11 datasets.
Revised: Figure 2(a) depicts the relative improvement of our proposed strategy when compared to CoOp in terms of the harmonic mean, where we observe an improvement in 10 out of 11 datasets.

Original: The limited generalization capability of CoOp is mitigated by CoCoOp by exploiting instance-conditional prompts, which improves the accuracy on new classes from 63 .
Revised: The limited generalization capability of CoOp is mitigated by CoCoOp by exploiting instance-conditional prompts, which improves the accuracy on new classes from 63 .

Original: 22% to 72 .
Revised: 22% to 72 .

Original: Nonetheless, augmenting
Revised: Nonetheless, augmenting

Original: CoCoOp with variational prompts still improves performance on the new classes and the harmonic mean by 3 .
Revised: CoCoOp with variational prompts still improves performance on the new classes and the harmonic mean by 3 .

Original: 25% and 1 .
Revised: 25% and 1 .

Original: 6% , respectively, with a mere decrease of 0 .
Revised: 6% , respectively, with a small decrease of 0 .

Original: 37% in base accuracy.
Revised: 37% in base accuracy.

Revised: Note here however that training for longer increases the performance on base classes and lowers it on new classes.

Revised: Our setting optimizes harmonic mean, but a simple tweak to the training scheduler on

Revised: EuroSAT is enough to surpass CoCoOp on base class accuracy, see Table 8 on appendix A.2.2.

Original: Figure 2(b) shows per-dataset relative harmonic mean improvement.
Revised: Figure 2(b) shows per-dataset relative harmonic mean improvement.

Original: We observe an improvement in 10 out of 11 datasets.
Revised: We observe an improvement inout of 11 datasets.

Original: Note that the drop in the average base accuracy for CoCoOp+VPT is negligible compared with CoOp+VPT .
Revised: Note that the drop in the average base accuracy for CoCoOp+VPT is negligible compared with CoOp+VPT .

Original: Moreover, our best performing model CoCoOp+VPT performs better than ProDA (Lu et al., 2022) in terms of new classes accuracy and harmonic mean by %2 .
Revised: Moreover, our best performing model CoCoOp+VPT performs better than ProDA (Lu et al., 2022) in terms of new classes accuracy and harmonic mean by %2 .

Original: 64 and %0 .78 .
Revised: 64 and %0 .

Revised: We also considered ensemble learning the soft prompt from different ways of initialization, this does not lead to an improvement (see Appendix A.2.3).

Original: It is also worth noting that CoOp+VPT and CoCoOp+VPT excel at CLIP zero-shot performance in base, new and harmonic accuracies.
Revised: It is also worth noting that CoOp+VPT and CoCoOp+VPT excel at CLIP zero-shot performance in base, new and harmonic accuracies.

Original: 4. C ROSS - DATASET TRANSFER LEARNING
Revised: 4. C ROSS - DATASET TRANSFER LEARNING

Original: Setup.
Revised: Setup.

Original: For cross-dataset transfer learning, the model is trained on a source dataset (ImageNet) and then assessed on 10 distinct target datasets.
Revised: For cross-dataset transfer learning, the model is trained on a source dataset (ImageNet) and then assessed on 10 distinct target datasets.

Original: This experiment tries to determine how effectively our methods generalizes transfer beyond the scope of a single dataset.
Revised: This experiment tries to determine how effectively our methods generalizes transfer beyond the scope of a single dataset.

Original: Results.
Revised: Results.

Original: As reported in Table 2, CoOp+VPT has a drop in performance in ImageNet by 1 .
Revised: As reported in Table 2, CoOp+VPT has a drop in performance in ImageNet by 1 .

Original: 76% while outperforming the target dataset on average by 1 . 63% .
Revised: 76% while outperforming the target dataset on average by 1 . 63% .

Original: Moreover, our proposed method leads to an increase on 9 out of 10 target datasets, with a small drop in accuracy of 0 .
Revised: Moreover, our proposed method leads to an increase on 9 out of 10 target datasets, with a small drop in accuracy of 0 .

Original: 03% in Caltech101.
Revised: 03% in Caltech101.

Original: Note[Table] that on target datasets such as FGVCAircraft and DTD
Revised: Note that on target datasets such as FGVCAircraft and DTD, our proposed method achieves an improvement of more than 3% .

Revised: Similarly to CoOp, augmenting CoCoOp with our method still leads to an

Original: [Table]
Revised: [Table]

Original: our proposed method achieves an improvement of more than 3% . Similarly to CoOp, augmenting CoCoOp with our method still leads to an overall performance enhancement of 0 .
Revised: overall performance enhancement of 0 .

Original: 16% on 7 out of 10 target datasets, showing its effectiveness for cross-dataset transfer learning.
Revised: 16% on 7 out of 10 target datasets, showing its effectiveness for cross-dataset transfer learning.

Original: In addition, unlike CoCoOp , which has better performance in
Revised: In addition, unlike CoCoOp , which has better performance in

Original: ImageNet-like datasets such as Caltech101 and OxfordPets, our proposed method exhibits improvement on dissimilar datasets (e.g. FGVCAircraft, DTD, and EuroSAT), demonstrating its capacity to capture the unique characteristics of each dataset.
Revised: ImageNet-like datasets such as Caltech101 and OxfordPets, our proposed method exhibits improvement on dissimilar datasets (e.g. FGVCAircraft, DTD, and EuroSAT), demonstrating its capacity to capture the unique characteristics of each dataset.

Original: 4. D OMAIN GENERALIZATION
Revised: 4. C ROSS - DOMAIN G ENERALIZATION

Original: Setup.
Revised: Setup.

Original: Lastly, We examine variational prompt tuning through the lens of distribution shift and robustness.
Revised: Lastly, We examine variational prompt tuning through the lens of distribution shift and robustness.

Original: We train our proposed model on the source dataset (ImageNet) for three different random seeds, and assess it on ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R. Prior work 
Revised: We train our proposed model on the source dataset (ImageNet) for three different random seeds, and assess it on ImageNetV2, ImageNet-Sketch, ImageNet-A, and ImageNet-R. Prior work such as CoOp (Zhou et al., 2022b) and

Revised: CoCoOp (Zhou et al., 2022a) demonstrate empirically that learning a soft-prompt improves the model’s resilience against distribution shift and adversarial

Original: [Table]
Revised: [Table]

Original: +
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: 99 + 0.11 + 0.37 + 0.50 + 0.46 + 0. such as CoOp (Zhou et al., 2022b) and CoCoOp (Zhou et al., 2022a) demonstrate empirically that learning a soft-prompt improves the model’s resilience against distribution shift and adversarial attack.
Revised: attack.

Original: Following their experiments, we are also interested in determining if treating prompts in a variational manner maintain or improve the performance.
Revised: Following their experiments, we are also interested in determining if treating prompts in a variational manner maintain or improve the performance.

Original: Results.
Revised: Results.

Original: As reported in Table 3 our method enhances the accuracy of CoOp on ImageNet-Sketch,
Revised: As reported in Table 3 our method enhances the accuracy of CoOp on ImageNet-Sketch,

Original: ImageNet-A, and ImageNet-R by 0 .
Revised: ImageNet-A, and ImageNet-R by 0 .

Original: 70% , and 2 .
Revised: 70% , and 2 .

Original: 19% while degrading the performance on
Revised: 19% while degrading the performance on

Original: ImageNet and ImageNetV2 by 1 . 78% , by 1 . 03% .
Revised: ImageNet and ImageNetV2 by 1 . 78% , by 1 . 03% .

Original: However, on CoCoOp, adding VPT, while losing the performance on source dataset similar to CoOp, consistently improves the accuracy on all target datasets which highlight the effectiveness of our proposed method.
Revised: However, on CoCoOp, adding VPT, while losing the performance on source dataset similar to CoOp, consistently improves the accuracy on all target datasets which highlight the effectiveness of our proposed method.

Original: [Table]
Revised: [Table]

Original: 4. A BLATIONS
Revised: 4. A BLATIONS

Original: Effectiveness of the posterior distribution q ϕ .
Revised: Effectiveness of the posterior distribution q ϕ .

Original: We first ablate the effectiveness of the variational posterior distribution.
Revised: We first ablate the effectiveness of the variational posterior distribution.

Original: To do this, we consider sampling one residual vector from the uniform distribution U ( 0 , 1 ) , normal distribution N ( 0 , I ) , normal distribution N ( µ ( x ) , 0 ) , normal distribution
Revised: To do this, we consider sampling one residual vector from the uniform distribution U ( 0 , 1 ) , normal distribution N ( 0 , I ) , normal distribution N ( µ ( x ) , 0 ) , normal distribution

Original: N ( µ ( x ) , Σ( x )) , and report the new class accuracy for CoCoOp+VPT for one random seed in Table 4.
Revised: N ( µ ( x ) , Σ( x )) , and report the new class accuracy for CoCoOp+VPT for one random seed in Table 4.

Original: Except for the EuroSAT dataset, a sample from the normal distribution N ( µ ( x ) , 0 ) obtains the best performance in comparison with alternatives, showing that the mean of the normal distribution µ ( x ) is the most effective sample.
Revised: Except for the EuroSAT dataset, a sample from the normal distribution N ( µ ( x ) , 0 ) obtains the best-performingce in comparison with alternatives, showing that the mean of the normal distribution µ ( x ) is the most effective sample.

Original: In addition, we find that drawing one sample from
Revised: In addition, we find that drawing one sample from

Original: N ( µ ( x ) , Σ( x )) yields superior results compared to drawing one sample from uniform distribution
Revised: N ( µ ( x ) , Σ( x )) yields superior results compared to drawing one sample from uniform distribution

Original: U ( 0 , 1 ) and normal distribution N ( 0 , I ) , further demonstrating the efficacy of our proposed method in capturing the underlying distribution of the prompt space.
Revised: U ( 0 , 1 ) and normal distribution N ( 0 , I ) , further demonstrating the efficacy of our proposed method in capturing the underlying distribution of the prompt space.

Original: We also ablate increasing the number of samples from the normal distribution N ( µ ( x ) , Σ( x )) to understand the informativeness of the learned variational distribution.
Revised: We also ablate increasing the number of samples from the normal distribution N ( µ ( x ) , Σ( x )) to understand the informativeness of the learned variational distribution.

Original: It is shown that enlarging the number of samples further improves the model performance as they capture the prompt space appropriately.[Table]
Revised: It is shown that enlarging the number of samples further improves the model performance as they capture the prompt space appropriately.

Original: Prompt initialization.
Revised: Prompt initialization.

Original: We investigate the effectiveness of the prompt initialization on the new class accuracy for CoCoOp+VPT for one random seed.
Revised: We investigate the effectiveness of the prompt initialization on the new class accuracy for CoCoOp+VPT for one random seed.

Original: We consider two variants.
Revised: We consider two variants.

Original: In the first one, we initialize the context tokens randomly using a normal distribution, whereas in second one we initialize the context tokens with “An image of a { class } ”.
Revised: In the first one, we ini- [Table] tialize the context tokens randomly using a normal distribution, whereas in second one we initialize the context tokens with “An image of a { class } ”.

Original: Table 5 summarizes this ablation.
Revised: Table 5 summarizes this ablation.

Original: Comparing the two variants demonstrates that an appropriately initialized prompt consistently outperforms a randomly initialized prompt, highlighting the necessity for further research of the prompt space.
Revised: Comparing the two variants demonstrates that an appropriately initialized prompt consistently outperforms a randomly initialized prompt, highlighting the necessity for further research of the prompt space.

Original: an appropriate prompt “An image of a { class } ” improves the performance compared to random tokens.

Original: Method DTD Flowers EuroSAT
Revised: We will leave it open for future research direction

Original: [Figure]
Revised: [Table]

Original: Random 74.

Original: [Figure] [Figure] [Figure] “An image of a { class } ”
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Number of Monte-Carlo samples.
Revised: Number of Monte-Carlo samples.

Original: When approximating the log-likelihood of input data, the number of Monte
Revised: When approximating the log-likelihood of input data, the number of Monte

Original: Carlo samples is an important hyperparameter.
Revised: Carlo samples is an important hyperparameter.

Original: Generally, a large number of samples should lead to a better approximation and better classification accuracy.
Revised: Generally, a large number of samples should lead to a better approximation and better classification accuracy.

Original: We ablate this hyperparameter on new accuracy for CoCoOp+VPT by varying the number of Monte Carlo samples at inference time.
Revised: We ablate this hyperparameter on new accuracy for CoCoOp+VPT by varying the number of Monte Carlo samples at inference time.

Original: We show results for a varying number of samples in Figure 3 for DTD, Flowers102, EuroSAT,  and UCF101.
Revised: We show results for a varying number of samples in Figure 3 for DTD, Flowers102, EuroSAT, FGVCAircraft, and UCF101.

Original: Increasing the Monte Carlo samples fromto 10 consistently improves the new accuracy, afterwards the accuracy saturates.
Revised: Increasing the Monte Carlo samples from 1 to 10 consistently improves the new accuracy, afterwards the accuracy saturates.

Original: Hence, we recommend evaluating variational prompt tuning on a larger number of Monte Carlo samples for better model accuracy.
Revised: Hence, we recommend evaluating variational prompt tuning on a larger number of Monte Carlo samples for better model accuracy.

Revised: To further aid the interpretation and understanding of the learned prompts we provide a further ablation on the variational distribution of our best performing model,

Original: For more ablations, see Section A.2 in the appendix.
Revised: CoCoOp+VPT in Appendix A.2.4 and A.2.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduce variational prompt tuning, a probabilistic modeling of the underlying distribution of prompts, allowing prompts within the support of an associated concept to be derived through stochastic sampling.
Revised: In this paper, we introduce variational prompt tuning, a probabilistic modeling of the underlying distribution of prompts, allowing prompts within the support of an associated concept to be derived through stochastic sampling.

Original: By doing so, we are able to generate adaptive context tokens for each data point.
Revised: By doing so, we are able to generate adaptive context tokens for each data point.

Original: This formulation leads to better generalization capabilities in terms of the new accuracy and harmonic mean for downstream tasks.
Revised: This formulation leads to better generalization capabilities in terms of the new accuracy and harmonic mean for downstream tasks.

Original: We show that it can be seamlessly integrated into both standard and conditional prompt learning frameworks, considerably improving the performance in both cases.
Revised: We show that it can be seamlessly integrated into both standard and conditional prompt learning frameworks, considerably improving the performance in both cases.

Original: We conduct extensive experiments on 15 datasets and demonstrate the benefits of a variational formulation in learning data-driven prompts.
Revised: We conduct extensive experiments on 15 datasets and demonstrate the benefits of a variational formulation in learning data-driven prompts.

Original: Our method provides the current state-ofthe-art for prompt learning, and constitutes, to the best of our knowledge, the first method for CLIP adaptation that fully maintains the generalization capability to new classes of the original model.
Revised: Our method provides the current state-ofthe-art for prompt learning, and constitutes, to the best of our knowledge, the first method for CLIP adaptation that fully maintains the generalization capability to new classes of the original model.

Revised: Finally, prompt distribution can be obtained using other ways of density estimation, e.g. normalizing flows, energy-based models or diffusion models, and we will leave them open for future research.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: Our method has the potential to affect applications that often need rapid adaptation, such as medical imaging, astronomical imaging, and autonomous driving.
Revised: Our method has the potential to affect applications that often need rapid adaptation, such as medical imaging, astronomical imaging, and autonomous driving.

Original: Because of this, there may be negative societal consequences associated with the adoption of our technology.
Revised: Because of this, there may be negative societal consequences associated with the adoption of our technology.

Original: For example, a lack of fairness with models trained on insufficient data, regulatory compliance, patient privacy in medical imaging, and more general biases encoded into large vision-language models such as CLIP.
Revised: For example, a lack of fairness with models trained on insufficient data, regulatory compliance, patient privacy in medical imaging, and more general biases encoded into large vision-language models such as CLIP.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: Details on benchmarks, metrics, and the implementation of variational prompt tuning in terms of architecture and training are contained in Section 4.
Revised: Details on benchmarks, metrics, and the implementation of variational prompt tuning in terms of architecture and training are contained in Section 4.

Original: We also list of all hyperparameters in Tableand 7.
Revised: We also list of all hyperparameters in Tableand 7.

Original: Lastly, we will release the source code, scripts to reproduce the results and evaluating the performance of the model at: https://github.com/<redacted>/<redacted> .
Revised: Lastly, we will release the source code, scripts to reproduce the results and evaluating the performance of the model at: https://github.com/<redacted>/<redacted> .

Original: A A PPENDIX A. H YPERPARAMETERS
Revised: A A PPENDIX A. H YPERPARAMETERS

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we provide the detailed hyperparameter settings in Tables 6 and 7 that are used to generate results in the main paper for each dataset.
Revised: we provide the detailed hyperparameter settings in Tables 6 and 7 that are used to generate results in the main paper for each dataset.

Original: There are two sets of hyperameters.
Revised: There are two sets of hyperameters.

Original: The first are shared among the two variants of variational prompt tuning CoOp+VPT and CoCoOp+VPT (See
Revised: The first are shared among the two variants of variational prompt tuning CoOp+VPT and CoCoOp+VPT (See

Original: Table 6).
Revised: Table 6).

Original: The second correspond to dataset-specific parameters that are optimized per dataset (See
Revised: The second correspond to dataset-specific parameters that are optimized per dataset (See

Original: Table 7)
Revised: Table 7)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: I m ag e N e t
Revised: I m ag e N e t

Original: C a l t ec h
Revised: C a l t ec h

Original: O x f o r d P e t s
Revised: O x f o r d P e t s

Original: S t a n f o r d
Revised: S t a n f o r d

Original: C a r s
Revised: C a r s

Original: F l o w er s
Revised: F l o w er s

Original: F oo d
Revised: F oo d

Original: F G
Revised: F G

Original: VCA i rcr a ft
Revised: VCA i rcr a ft

Original: S U N
Revised: S U N

Original: D T D
Revised: D T D

Original: E u r o
Revised: E u r o

Original: S A T
Revised: S A T

Original: UC F
Revised: UC F

Original: [Table]
Revised: [Table]

Original: A. M ORE ABLATIONS
Revised: A. M ORE ABLATIONS

Original: Vision encoder alternatives.
Revised: A.2. V ISION ENCODER ALTERNATIVES

Original: All previous experiments benefit from ViT-B/16 as the vision encoder’s backbone following (Zhou et al., 2022b;a; Lu et al., 2022). 
Revised: All previous experiments benefit from ViT-B/16 as the vision encoder’s backbone following (Zhou et al., 2022b;a; Lu et al., 2022). For completeness, in Figures 4,

Original: For completeness, in Figures 4, 5, and 6, we replace this vision encoder with a Resnet50 and Resnet100 and examine its impact on base accuracy, new accuracy and harmonic mean for one random seed.
Revised: [Figure] and [Figure] we replace this vision encoder with a Resnet50 and Resnet100 and examine its impact on base accuracy, new accuracy and harmonic mean for one random seed.

Original: The visual transformer outperforms the Resnet alternatives across all 10 datasets for new accuracy and harmonic mean and inout of 10 datasets for base accuracy.
Revised: The visual transformer outperforms the Resnet alternatives across all 10 datasets for new accuracy and harmonic mean and in 9 out of 10 datasets for base accuracy.

Original: Hence, we suggest training and evaluating variational prompt tuning on visual transformer for better model performance 
Revised: Hence, we suggest training and evaluating variational prompt tuning on visual transformer for better model performance [Figure] [Figure] [Figure]

Revised: to harmonic mean .

Original: [Figure]
Revised: A more over-parameterized model leads to better performance across all datasets. A.2. B ASE AND NEW CLASSES ACCURACY TRADE - OFF In this section [Table]

Revised: we ablate whether there is a trade-off between base and new classes accuracy.

Revised: To do so, we train our proposed method on EuroSAT dataset for 60 and 80 epochs, and compare them with in Table 8.

Revised: As shown, training EuroSAT for 20 more epochs raises the performance on base classes by +5 .

Revised: This alone improves the average performance on base classes across the 11 datasets by +0 . 49 , resulting in 80 .

Revised: 58 (now +0.11 over CoCoOp).

Revised: This change slightly affects average harmonic mean, reducing it by 0 .

Revised: 17% (now +1 .

Revised: 43% better than CoCoOp).

Revised: Consequently, we indeed observe a trade-off between performance on base and on new classes.

Revised: Training for longer will increase the performance on base classes and lower it on new classes.

Revised: [Table]

Revised: E NSEMBLE BASELINES

Revised: Variational prompt tuning can be considered as an efficient ensemble approach as it generates several samples from the prompt space and uses them in conjunction.

Revised: Nonetheless, we compared CoCoOp+VPT against a modified CoCoOp that naively uses an ensemble of prompts.

Revised: We implement this by initializing several learnable prompts per class using M hand-crafted templates provided in the CLIP codebase 2 and fine-tuning them together.

Revised: The final text feature representation is computed as a simple average of the ensemble of prompts for each class as in Section 3.1.4 of (Radford et al., 2021).

Revised: For a fair comparison, the number of ensembles, M , equates to the number of Monte Carlo samples K per dataset (see Table 7).

Revised: In Table 9, when comparing CoCoOp and CoCoOp + Ens, the ensemble counterpart, we can see that adding an ensemble of prompts does not necessarily lead to an improvement in all datasets.

Revised: For instance, on DTD and UCF101 datasets, CoCoOp performs better, while on Flowers102, EuroSAT, and FGVCAircraft datasets, CoCoOp + Ens has a higher harmonic mean.

Revised: Moreover, when comparing CoCoOp + Ens against CoCoOp+VPT , we can see that our method performs better in terms of new classes accuracy and harmonic mean in all datasets other than Flowers102, with an average increase of 1.94% in harmonic mean, whereas the drop in

Revised: Flower102 is only 0.66%.

Revised: A.2. F ACTOR OF VARIATION ANALYSIS

Revised: Here, we analyze whether prompts that are sampled at the distributional modes correlate with features that characterize any known factors of variation (e.g., sub-domains) in a dataset.

Revised: We perform this experiment on our best-performing model, CoCoOp+VPT.

Revised: First, we randomly select three classes c 1 , c 2 , and c 3 from the Flowers102 dataset (Columbine,

Revised: Passion Flower and Cyclamen, for reference).

Revised: Then, for each class, we compute image features by forwarding the respective input images x into the image encoder of the CLIP.

Revised: Next, we apply

Revised: K-means to group the image features into Q = 5 cluster centroids.

Revised: Note that each cluster centroid

Revised: [Table]

Revised: is assumed to capture some factors of visual variation within each class.

Revised: Afterwards, we treat the cluster assignments as pseudo-labels of those images.

Revised: Note that this is done separately for each class.

Revised: Additionally, for an input image x , we generate K = 21 different prompts.

Revised: These prompts are fed into the clip’s text encoder

Original: [Figure]
Revised: [Table] which generates K different weights w 1 , w 2 , · · · , w K per image instance We now have a set of K weights for Q pseudo-labels across three classes, which are visualized in each row of Figure 7. In each row, on the left side, we visualize how prompt samples are distributed across the five different clusters, while on the right side, we provide the top-3 most representative samples of the individual clusters (i.e. the images that produce feature representations that are the closets to the centroids).

Revised: From this figure, it can be seen that there is a high intersection between the distribution of all sub classes, representing shared knowledge.

Revised: However, some particular visual differences are expressed in regions without intersection in the distribution.

Revised: Consequently, we believe that there is a correlation between the modes of the variational distribution and visual particularities of subsets of the same class.

Original: [Figure]
Revised: [Figure]

Revised: A.2. I NTERPRETATION OF VARIATIONAL DISTRIBUTION

Revised: We now provide an intuition of how prompt samples are distributed within the variational distribution.

Revised: We perform two experiments on our best-performing model, CoCoOp+VPT.

Revised: Experiment 1: For an input image x and its corresponding target y , K residuals are sampled from the conditional distribution π ϕ ( x ) , which are used to generate K different prompts per class p k =[ p 1 + r k , p 2 + r k , · · · , p L + r k ] .

Revised: We also construct a new prompt based on the mean of variational distribution µ ( x ) named as mean prompt p µ ( x ) =[ p 1 + µ ( x ) , p 2 + µ ( x ) , · · · , p L + µ ( x )] .

Revised: These K prompts are sorted based on their distance to the mean of variational distribution µ ( x ) , where for any i and j ( i ≤ j ), d ( µ ( x ) , r i ) ≤ d ( µ ( x ) , r j ) .

Revised: The K prompts and mean prompt are prepended to the class-specific embedding to generate a series of K + 1 separate classifier weights w µ ( x ) ,c , w 1 ,c , w 2 ,c , · · · , w K,c .

Revised: For each class y , we compute the cosine similarity of the image encoding f

Revised: ( x ) and w µ ( x ) ,y , w 1 ,y , w 2 ,y , · · · , w K,y and average across all samples per class.

Revised: In

Revised: Figure 8, we can see that the mean prompt p µ ( x ) is the most similar classifier weight to the image encoding , and, as we move further away from the mean prompt, the cosine similarity decreases .

Revised: Experiment 2:

Revised: Given weights w µ ( x ) ,y , w 1 ,y , w 2 ,y , · · · , w K,y from Experiment 1, we define a new weight as w Jy = w µ ( x ) ,y + (cid:80)

Revised: Jj =1 w j,y , where w Jy is the cumulative sum of weights regarding prompt p 1 to p J and mean prompt p µ ( x ) .

Revised: The cosine similarity between the image encoding f ( x ) and all { w jy } j = K j =1 are computed and visualized in Figure 8.

Original: over-parameterized model leads to better performance across all datasets.
Revised: As shown, summing up classifier’s weights together increases the cosine similarity , with the maximum similarity obtained when all samples are combined . From these two experiments, we believe that prompt samples are well-distributed inside the prompt space such that the prompt distribution provides adequate coverage of the underlying distribution for downstream tasks. [Figure] [Figure]