Original: P ROMPT T UNING WITH P ROMPT - ALIGNED G RADIENT FOR V ISION -L ANGUAGE M ODELS
Revised: P ROMPT T UNING WITH P ROMPT - ALIGNED G RADIENT FOR V ISION -L ANGUAGE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Thanks to the large pre-trained vision-language models (VLMs) like CLIP (Radford et al., 2021), we can craft a zero-shot classifier by discrete prompt design, e.g. , the confidence score of an image being “ [CLASS] ” can be obtained by using the VLM provided similarity between the image and the prompt sentence “ a photo of a [CLASS] ”.
Revised: Thanks to the large pre-trained vision-language models (VLMs) like CLIP (Radford et al., 2021), we can craft a zero-shot classifier by discrete prompt design, e.g. , the confidence score of an image being “ [CLASS] ” can be obtained by using the VLM provided similarity between the image and the prompt sentence “ a photo of a [CLASS] ”.

Original: Furthermore, prompting shows great potential for fast adaptation of VLMs to downstream tasks if we fine-tune the soft prompts with few samples.
Revised: Furthermore, prompting shows great potential for fast adaptation of VLMs to downstream tasks if we fine-tune the soft prompts with few samples.

Original: However, we find a common failure that improper fine-tuning or learning with extremely few-shot samples may even under-perform the zeroshot prediction.
Revised: However, we find a common failure that improper fine-tuning or learning with extremely few-shot samples may even under-perform the zeroshot prediction.

Original: Existing methods still address this problem by using traditional anti-overfitting techniques such as early stopping and data augmentation, which lack a principled solution specific to prompting.
Revised: Existing methods still address this problem by using traditional anti-overfitting techniques such as early stopping and data augmentation, which lack a principled solution specific to prompting.

Original: In this paper, we present Promptaligned Gradient, dubbed ProGrad to prevent prompt tuning from forgetting the general knowledge learned from VLMs.
Revised: In this paper, we present Promptaligned Gradient, dubbed ProGrad to prevent prompt tuning from forgetting the general knowledge learned from VLMs.

Original: In particular, ProGrad only updates the prompt whose gradient is aligned (or non-conflicting) to the general knowledge, which is represented as the optimization direction offered by the pre-defined prompt predictions.
Revised: In particular, ProGrad only updates the prompt whose gradient is aligned (or non-conflicting) to the general knowledge, which is represented as the optimization direction offered by the pre-defined prompt predictions.

Original: Extensive experiments demonstrate the stronger few-shot generalization ability of ProGrad over state-of-the-art prompt tuning methods.
Revised: Extensive experiments demonstrate the stronger few-shot generalization ability of ProGrad over state-of-the-art prompt tuning methods.

Original: Codes and theoretical proof are in Appendix.
Revised: Codes and theoretical proof are in Appendix.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: After seeing and reading countless image-text association pairs, large and deep vision-language models (VLM) (Radford et al., 2021; Jia et al., 2021) can memorize the general knowledge (a.k.a.
Revised: After seeing and reading countless image-text association pairs, large and deep vision-language models (VLM) (Radford et al., 2021; Jia et al., 2021) can memorize the general knowledge (a.k.a.

Original: encyclopedic knowledge) about what visual patterns correspond to what textual sequence and vice versa.
Revised: encyclopedic knowledge) about what visual patterns correspond to what textual sequence and vice versa.

Original: Thanks to the powerful language modeling of VLMs, we can establish a communication channel in human-readable natural language, i.e. , prompt (Liu et al., 2021a; Yao et al., 2021; Jin et al., 2022), to query the general knowledge.
Revised: Thanks to the powerful language modeling of VLMs, we can establish a communication channel in human-readable natural language, i.e. , prompt (Liu et al., 2021a; Yao et al., 2021; Jin et al., 2022), to query the general knowledge.

Original: Prompting bridges the interface gap between the pre-trained and downstream tasks ( e.g. , regression vs. classification) without the need for additional fine-tuning adaptation.
Revised: Prompting bridges the interface gap between the pre-trained and downstream tasks ( e.g. , regression vs. classification) without the need for additional fine-tuning adaptation.

Original: For example, we can craft a concrete prompt—“ a photo of a [CLASS] ”—to achieve zero-shot image classification: by using the popular vision-language model CLIP (Radford et al., 2021), we input the image to the vision end and the prompt sentence to the language end, then obtain a vision-language similarity as the confidence score of classifying the image as “ [CLASS] ”.
Revised: For example, we can craft a concrete prompt—“ a photo of a [CLASS] ”—to achieve zero-shot image classification: by using the popular vision-language model CLIP (Radford et al., 2021), we input the image to the vision end and the prompt sentence to the language end, then obtain a vision-language similarity as the confidence score of classifying the image as “ [CLASS] ”.

Original: In practice, the prompt-based zero-shot image classification is not accurate because the hand-crafted prompt may not be the most machine-favorable ( e.g. , “ this is a picture of ” could be more grammatically prevailing in VLM training), or not specific to the downstream domain ( e.g. , “ a photo of a person doing ” is better in action recognition)
Revised: In practice, the prompt-based zero-shot image classification is not accurate because the hand-crafted prompt may not be the most machine-favorable ( e.g. , “ this is a picture of ” could be more grammatically prevailing in VLM training), or not specific to the downstream domain ( e.g. , “ a photo of a person doing ” is better in action recognition)

Original: (Radford et al., 2021).
Revised: (Radford et al., 2021).

Original: Recently, prompt tuning or prefix tuning (Lester et al., 2021;
Revised: Recently, prompt tuning or prefix tuning (Lester et al., 2021;

Original: Liu et al., 2021b;
Revised: Liu et al., 2021b;

Original: Zhou et al., 2021; 2022) has been proposed to replace the hand-crafted prompt with a set of tunable word embedding vectors, which do not have to be translatable back to human-readable words.
Revised: Zhou et al., 2021; 2022) has been proposed to replace the hand-crafted prompt with a set of tunable word embedding vectors, which do not have to be translatable back to human-readable words.

Original: Yet, prompt tuning is still as tricky as conventional fine-tuning: as the training continues, the generalization ability may decrease and even under-perform the zero-shot baseline.
Revised: Yet, prompt tuning is still as tricky as conventional fine-tuning: as the training continues, the generalization ability may decrease and even under-perform the zero-shot baseline.

Original: As shown in
Revised: As shown in

Original: Figure 1(a&b), the prompt tuning method CoOp (Zhou et al., 2021) achieves the best results via early stopping, and its accuracies heavily drop by at most 4% when the training continues.
Revised: Figure 1(a&b), the prompt tuning method CoOp (Zhou et al., 2021) achieves the best results via early stopping, and its accuracies heavily drop by at most 4% when the training continues.

Original: Besides,
Revised: Besides,

Original: Figure 1(c&d) show that CoOp underperforms zero-shot CLIP without augmentation or enough samples from downstream tasks.
Revised: Figure 1(c&d) show that CoOp underperforms zero-shot CLIP without augmentation or enough samples from downstream tasks.

Original: To the best of our knowledge, existing methods still rely on the
Revised: To the best of our knowledge, existing methods still rely on the

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: T L S e d a n
Revised: T L S e d a n

Original: [Figure]
Revised: [Figure]

Original: ud
Revised: ud

Original: i
Revised: i

Original: R S
Revised: R S

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: conventional anti-overfitting techniques such as early stopping and data augmentation (Zhou et al., 2021; 2022; Gao et al., 2021b; Qin & Joty, 2022a), which lacks a principled solution to the nature of improper prompt tuning.
Revised: conventional anti-overfitting techniques such as early stopping and data augmentation (Zhou et al., 2021; 2022; Gao et al., 2021b; Qin & Joty, 2022a), which lacks a principled solution to the nature of improper prompt tuning.

Original: Furthermore, the Grad-CAM visualization results indicate that the fine-tuned prompt starts to mislead the VLM to forget the general knowledge that the classification should at least focus on the foreground object but not the background.
Revised: Furthermore, the Grad-CAM visualization results indicate that the fine-tuned prompt starts to mislead the VLM to forget the general knowledge that the classification should at least focus on the foreground object but not the background.

Original: Comparing CoOp (Figure 2(b)) with zero-shot CLIP (Figure 2(c)), we find that the CoOp model distracts its attention to the background, while CLIP mainly focuses on the foreground object.
Revised: Comparing CoOp (Figure 2(b)) with zero-shot CLIP (Figure 2(c)), we find that the CoOp model distracts its attention to the background, while CLIP mainly focuses on the foreground object.

Original: These results demonstrate the over-fitting risk of existing prompt tuning strategies, especially when the number of training samples is extremely limited ( e.g. , 1 or 2).
Revised: These results demonstrate the over-fitting risk of existing prompt tuning strategies, especially when the number of training samples is extremely limited ( e.g. , 1 or 2).

Original: To this end, we present a novel prompt tuning method called Prompt-aligned Gradient ( ProGrad ) to overcome the improperly biased tuning for CLIP.
Revised: To this end, we present a novel prompt tuning method called Prompt-aligned Gradient ( ProGrad ) to overcome the improperly biased tuning for CLIP.

Original: The principle of ProGrad is to regularize each tuning step not to conflict with the general knowledge offered by the original prompt, e.g. , the zero-shot CLIP predictions.
Revised: The principle of ProGrad is to regularize each tuning step not to conflict with the general knowledge offered by the original prompt, e.g. , the zero-shot CLIP predictions.

Original: Specifically, we measure the general knowledge direction G g using the gradient of Kullback–Leibler (KL) divergence between the predictions of the zero-shot prompted
Revised: Specifically, we measure the general knowledge direction G g using the gradient of Kullback–Leibler (KL) divergence between the predictions of the zero-shot prompted

Original: CLIP and the few-shot fine-tuned model, which we name as general direction .
Revised: CLIP and the few-shot fine-tuned model, which we name as general direction .

Original: Similarly, we compute the domain-specific knowledge direction G d using the gradient of cross-entropy between the groundtruth and the few-shot fine-tuned model, dubbed domain-specific direction.
Revised: Similarly, we compute the domain-specific knowledge direction G d using the gradient of cross-entropy between the groundtruth and the few-shot fine-tuned model, dubbed domain-specific direction.

Original: We decompose the domain-specific direction G d into: 1) a vector G ⊥ orthogonal to the general direction, which denotes the non-conflicting domain-specific knowledge; and 2) a vector G ∥ parallel to the general direction, which denotes the general knowledge.
Revised: We decompose the domain-specific direction G d into: 1) a vector G ⊥ orthogonal to the general direction, which denotes the non-conflicting domain-specific knowledge; and 2) a vector G ∥ parallel to the general direction, which denotes the general knowledge.

Original: Note that the first gradient component does NOT override the general direction as any two orthogonal vectors can be transformed into two non-conflicting base vectors.
Revised: Note that the first gradient component does NOT override the general direction as any two orthogonal vectors can be transformed into two non-conflicting base vectors.

Original: For the second component, it must be one of the two directions: 1) the same of the general direction, which indicates that the update is aligned to the general knowledge, and 2) the opposite of general direction, indicating a conflicting update that should be discarded to avoid forgetting.
Revised: For the second component, it must be one of the two directions: 1) the same of the general direction, which indicates that the update is aligned to the general knowledge, and 2) the opposite of general direction, indicating a conflicting update that should be discarded to avoid forgetting.

Original: Overall, in each iteration, ProGrad only updates the parameters in the prompt-aligned direction that has an acute angle to the general direction.
Revised: Overall, in each iteration, ProGrad only updates the parameters in the prompt-aligned direction that has an acute angle to the general direction.

Original: Compared to CoOp and CLIP, both G g and G ⊥ (Figure 2(d&e)) help to regularize the model to focus on the foreground, and our ProGrad (Figure 2(f)) can further improve the visual response.
Revised: Compared to CoOp and CLIP, both G g and G ⊥ (Figure 2(d&e)) help to regularize the model to focus on the foreground, and our ProGrad (Figure 2(f)) can further improve the visual response.

Original: Following CLIP, CoOp and CoCoOp (Zhou et al., 2022), we evaluate our ProGrad under the few-shot learning, domain generalization and base-to-new generalization  settings over 11 image classification benchmark datasets, covering generic object classification, fine-grained image recognition, action classification.
Revised: Following CLIP, CoOp and CoCoOp (Zhou et al., 2022), we evaluate our ProGrad under the few-shot learning, domain generalization, base-to-new generalization and cross-dataset transfer settings over 11 image classification benchmark datasets, covering generic object classification, fine-grained image recognition, action classification.

Original: In summary, our ProGrad achieves: 1) clear improvement compared to CoOp over all of the 11 datasets; 2) clear improvement on the harmonic mean of base-class and new-class accuracies on all 11 datasets compared to CoOp and CoCoOp, and 3) clear improvement on both the source and target datasets of the domain generalization.
Revised: In summary, our ProGrad achieves: 1) clear improvement compared to CoOp over all of the 11 datasets; 2) clear improvement on the harmonic mean of base-class and new-class accuracies on all 11 datasets compared to CoOp and CoCoOp, and 3) clear improvement on both the source and target datasets of the domain generalization.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Vision-Language Pre

Original: -training.

Original: The pre-text task for pre-training Vision-Language Models (VLMs) mainly falls into 4 categories: 1) Masked language modeling (Kim et al., 2021;

Original: Lu et al., 2019), 2)

Original: Masked region prediction (Tan & Bansal, 2019;

Original: Su et al., 2019), 3) Image-text matching (Tan &

Original: Bansal, 2019; Kim et al., 2021), and 4) Contrastive learning (Radford et al., 2021; Jia et al., 2021; Li et al., 2021; Huo et al., 2021).

Original: In our work, we focus on VLMs pre-trained by contrastive learning to align visual and textual embedding.

Original: CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) are the typical models that leverage large-scale image-text pairs to learn transferable visual representations and exhibit impressive prompt-based zero-shot performance on various image classification tasks.

Original: Fine-tuning for VLMs.
Revised: Fine-tuning for VLMs.

Original: It is for the VLM adaptation to various downstream tasks, e.g. , visual question answering (Kim et al., 2021; Tan & Bansal, 2019), visual grounding (Yao et al., 2021; Lu et al., 2019), image retrieval (Kim et al., 2021; Lu et al., 2019) and image classification (Zhou et al., 2021; 2022).
Revised: It is for the VLM adaptation to various downstream tasks, e.g. , visual question answering (Kim et al., 2021; Tan & Bansal, 2019), visual grounding (Yao et al., 2021), image retrieval (Lu et al., 2019), semantic segmentation (Rao et al., 2021) and image classification (Zhou et al., 2021; 2022).

Original: We focus on image classification task.
Revised: We focus on image classification task.

Original: Conventional “pre-train then fine-tune” paradigm that plugs in an additional classifier on top of visual backbone and trained on downstream data has been widely-adopted, e.g. , Linear Probe (Radford et al., 2021).
Revised: Conventional “pre-train then fine-tune” paradigm that plugs in an additional classifier on top of visual backbone and trained on downstream data has been widely-adopted, e.g. , Linear Probe (Radford et al., 2021).

Original: CLIP-Adapter (Gao et al., 2021a) and Tip-Adapter (Zhang et al., 2021) add vision and language feature adapter to boost conventional fine-tuning results.
Revised: CLIP-Adapter (Gao et al., 2021a) and Tip-Adapter (Zhang et al., 2021) add vision and language feature adapter to boost conventional fine-tuning results.

Original: Recently, NLP community presents a novel fine-tuning paradigm named “promptbased learning”, which is formulated as a “fill-in-the-blank” cloze test, and fine-tunes the prompt to maximize the ground-truth token (Lester et al., 2021; Liu et al., 2021b).
Revised: Recently, NLP community presents a novel fine-tuning paradigm named “prompt-based learning”, which is formulated as a “fill-in-the-blank” cloze test, and finetunes the prompt to maximize the ground-truth token (Lester et al., 2021; Liu et al., 2021b).

Original: In CV community, CoOp (Zhou et al., 2021) uses a continuous prompt optimization from downstream data instead of hand-craft design.
Revised: In CV community, CoOp (Zhou et al., 2021) uses a continuous prompt optimization from downstream data instead of hand-craft design.

Original: CoCoOp (Zhou et al., 2022) further extends CoOp by learning image conditional prompt rather than a static one to improve generalization to unseen classes.
Revised: CoCoOp (Zhou et al., 2022) further extends CoOp by learning image conditional prompt rather than a static one to improve generalization to unseen classes.

Original: DenseCLIP (Rao et al., 2021) applies context-aware prompt strategy to dense prediction tasks, e.g. , segmentation and object dectection.
Revised: ProDA (Lu et al., 2022) adapts VLMs to downstream classification tasks by learning a prompt distribution over the output embedding space. VPT (Derakhshani et al., 2022) introduces variational prompt tuning by combining a base learned prompt with a residual vector sampled from a instance-specific underlying distribution.

Original: Our proposed ProGrad follows the line of prompt-based learning to improve both few-shot classification performance and generalization ability by aligning the gradient to general direction, withoutextra model structure modification or tuning the pre-trained model parameters.
Revised: Our proposed ProGrad follows the line of prompt-based learning to improve both few-shot classification performance and generalization ability by aligning the gradient to general direction, without model structure modification or tuning the pre-trained model parameters.

Original: Knowledge Transfer.
Revised: Knowledge Transfer.

Original: Forgetting mitigation by knowledge distillation or memory replay is widely deployed in incremental learning (Liu et al., 2020; Rebuffi et al., 2017; Qin & Joty, 2022b;
Revised: Forgetting mitigation by knowledge distillation or memory replay is widely deployed in incremental learning (Liu et al., 2020; Rebuffi et al., 2017; Qin & Joty, 2022b;

Original: Riemer et al., 2018;
Revised: Riemer et al., 2018;

Original: Hu et al., 2021).
Revised: Hu et al., 2021).

Original: However, prompt-based fine-tuning is fundamentally different from incremental learning: the former assumes that VLMs have already captured all the knowledge needed in downstream tasks and the goal is to compose a domain-specific query, whereas the latter assumes that the knowledge is yet to be sufficient.
Revised: However, prompt-based fine-tuning is fundamentally different from incremental learning: the former assumes that VLMs have already captured all the knowledge needed in downstream tasks and the goal is to compose a domain-specific query, whereas the latter assumes that the knowledge is yet to be sufficient.

Original: In addition, incremental learning requires old data from memory storage while our prompt-based learning method has no access to the pretrained data.
Revised: In addition, incremental learning requires old data from memory storage while our prompt-based learning method has no access to the pre-trained data.

Revised: For example, OGD (Farajtabar et al., 2020) projects the gradients from new classes to the orthogonal direction of the gradients of previous tasks.

Revised: However, as we have no access to pre-training process, the requirement of OGD to store the gradients of old tasks is not possible for prompt tuning.

Revised: Moreover, OGD alters the gradients of downstream tasks even in non-conflicting scenarios which potentially results in sub-optimal performance for downstream tasks.

Original: Another related field that leverages gradient matching to transfer knowledge is domain generalization (Shi et al., 2022; Rame et al., 2021) and multi-task learning (Sener & Koltun, 2018; Yu et al., 2020).
Revised: Another related field that leverages gradient matching to transfer knowledge is domain generalization (Shi et al., 2022; Rame et al., 2021) and multi-task learning (Sener & Koltun, 2018; Yu et al., 2020).

Original: However, their methods are not directly applicable in prompt tuning whose transfer direction is only from general to downstream.
Revised: However, their methods are not directly applicable in prompt tuning whose transfer direction is only from general to downstream.

Original: In Appendix, we will show how their methods fail in several ablative studies.
Revised: In Appendix, we will show how their methods fail in several ablative studies.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In this section, we introduce the preliminary concepts of hand-crafted prompt-based zero-shot inference, prompt-based learning, and present our proposed Prompt-aligned Gradient solution to align the domain knowledge with general knowledge for few-shot generalization.
Revised: In this section, we introduce the preliminary concepts of hand-crafted prompt-based zero-shot inference, prompt-based learning, and present our proposed Prompt-aligned Gradient solution to align the domain knowledge with general knowledge for few-shot generalization.

Original: [Figure]
Revised: [Figure]

Original: 3. P RELIMINARIES
Revised: 3. P RELIMINARIES

Original: Contrastive language-image pre-training (CLIP) (Radford et al., 2021) adopts a contrastive language-image pre-training paradigm on tremendous pairs of images with natural language descriptions.
Revised: Contrastive language-image pre-training (CLIP) (Radford et al., 2021) adopts a contrastive language-image pre-training paradigm on tremendous pairs of images with natural language descriptions.

Original: For contrastive learning, the associated image and sentences are taken as the positive samples, while the non-associated pairs are regarded as negative samples.
Revised: For contrastive learning, the associated image and sentences are taken as the positive samples, while the non-associated pairs are regarded as negative samples.

Original: The contrastive objective maximizes the similarity of positive pairs while minimize the similarity of negative pairs.
Revised: The contrastive objective maximizes the similarity of positive pairs while minimize the similarity of negative pairs.

Original: Zero-shot transfer inference adapts the pre-trained CLIP model to downstream tasks without fine-tuning the model.
Revised: Zero-shot transfer inference adapts the pre-trained CLIP model to downstream tasks without fine-tuning the model.

Original: Taking image classification as an example, zero-shot transfer is enabled by formulating the classification task as an image-text matching problem, where the text is obtained by extending the “ [CLASS] ” name using a template like “ a photo of a [CLASS]. ”.
Revised: Taking image classification as an example, zero-shot transfer is enabled by formulating the classification task as an image-text matching problem, where the text is obtained by extending the “ [CLASS] ” name using a template like “ a photo of a [CLASS]. ”.

Original: CLIP (Radford et al., 2021) finds that such a simple template narrows the distribution gap to pre-training text inputs.
Revised: CLIP (Radford et al., 2021) finds that such a simple template narrows the distribution gap to pre-training text inputs.

Original: The image-class matching score is measured based on the cosine similarity < w i , f > between the image feature f and the class-extended text feature w i for i -th class.
Revised: The image-class matching score is measured based on the cosine similarity < w i , f > between the image feature f and the class-extended text feature w i for i -th class.

Original: The image feature f for image x is extracted by the image encoder, while the text feature w i for i -th class is obtained by feeding the prompt description into the text encoder.
Revised: The image feature f for image x is extracted by the image encoder, while the text feature w i for i -th class is obtained by feeding the prompt description into the text encoder.

Original: The probability for i -th class is obtained as
Revised: The probability for i -th class is obtained as

Original: [Equation]
Revised: [Equation]

Original: where K denotes the number of classes, and τ is a temperature learned by CLIP.
Revised: where K denotes the number of classes, and τ is a temperature learned by CLIP.

Original: Prompt-based learning further strengths the transferring ability of the CLIP model and avoids prompt engineering by automatically learning the prompt given few samples from the downstream task.
Revised: Prompt-based learning further strengths the transferring ability of the CLIP model and avoids prompt engineering by automatically learning the prompt given few samples from the downstream task.

Original: Different from the zero-shot transfer that used a fixed hand-craft prompt, CoOp (Zhou et al., 2021) constructs and fine-tunes a set of M continuous context vectors v = { v 1 , v 2 , ..., v M } as the turnable prompt.
Revised: Different from the zero-shot transfer that used a fixed hand-craft prompt, CoOp (Zhou et al., 2021) constructs and fine-tunes a set of M continuous context vectors v = { v 1 , v 2 , ..., v M } as the turnable prompt.

Original: Specifically, the prompt t i = { v 1 , v 2 , ..., v M , c i } combines the learnable context vectors v and the class token embedding c i , and is fed to the text encoder g ( · ) .
Revised: Specifically, the prompt t i = { v 1 , v 2 , ..., v M , c i } combines the learnable context vectors v and the class token embedding c i , and is fed to the text encoder g ( · ) .

Original: CoOp optimizes the static context vectors v by minimizing the negative log-likelihood of the ground-truth token:
Revised: CoOp optimizes the static context vectors v by minimizing the negative log-likelihood of the ground-truth token:

Original: [Equation]
Revised: [Equation]

Original: where y denotes the one-hot ground-truth annotation and K denotes the number of classes.
Revised: where y denotes the one-hot ground-truth annotation and K denotes the number of classes.

Original: 3. P ROMPT - ALIGNED G RADIENT
Revised: 3. P ROMPT - ALIGNED G RADIENT

Original: As we introduced in Section 1, CoOp faced a challenge that the transfer performance drops when the number of annotations is very limited ( e.g. , one per class), even underperforms the zero-shot transfer.
Revised: As we introduced in Section 1, CoOp faced a challenge that the transfer performance drops when the number of annotations is very limited ( e.g. , one per class), even underperforms the zero-shot transfer.

Original: Also, CoOp heavily relies on anti-overfitting techniques such as early stopping and data augmentation.
Revised: Also, CoOp heavily relies on anti-overfitting techniques such as early stopping and data augmentation.

Original: To overcome the over-fitting challenge, we propose an effective and efficient fine-tuning paradigm
Revised: To overcome the over-fitting challenge, we propose an effective and efficient fine-tuning paradigm

Original: ProGrad to align the few-shot downstream knowledge with the large-scale general knowledge.
Revised: ProGrad to align the few-shot downstream knowledge with the large-scale general knowledge.

Original: Motivated by the success of knowledge distillation (Phuong & Lampert, 2019; Hinton et al., 2015) in knowledge transfer, we leverage the zero-shot CLIP predictions as the general knowledge, and compare the fine-tuned predictions with the general knowledge to regularize the gradient direction.
Revised: Motivated by the success of knowledge distillation (Phuong & Lampert, 2019; Hinton et al., 2015) in knowledge transfer, we leverage the zero-shot CLIP predictions as the general knowledge, and compare the fine-tuned predictions with the general knowledge to regularize the gradient direction.

Original: Specifically, we obtain the domain-specific direction by calculating the cross-entropy L ce ( v ) between the model prediction p ( t i | x ) and the ground-truth y according to Eq.
Revised: Specifically, we obtain the domain-specific direction by calculating the cross-entropy L ce ( v ) between the model prediction p ( t i | x ) and the ground-truth y according to Eq.

Original: (2), and the general knowledge direction based on the Kullback-Leibler (KL) divergence between p ( t i | x ) and the zero-shot CLIP prediction p zs ( w i | x ) :
Revised: (2), and the general knowledge direction based on the Kullback-Leibler (KL) divergence between p ( t i | x ) and the zero-shot CLIP prediction p zs ( w i | x ) :

Original: [Equation]
Revised: [Equation]

Original: We denote the gradients of L kl ( v ) and L ce ( v ) as G g = ∇ v L kl ( v ) and G d = ∇ v L ce ( v ) , respectively.
Revised: We denote the gradients of L kl ( v ) and L ce ( v ) as G g = ∇ v L kl ( v ) and G d = ∇ v L ce ( v ) , respectively.

Original: The relations between G g and G d are two-fold.
Revised: The relations between G g and G d are two-fold.

Original: (1) Their angle is smaller than 90 ◦ (Figure 3(a)), which indicates that the optimization direction of few-shot downstream knowledge does not conflict with general knowledge.
Revised: (1) Their angle is smaller than 90 ◦ (Figure 3(a)), which indicates that the optimization direction of few-shot downstream knowledge does not conflict with general knowledge.

Original: In this case, we safely set the updated gradient direction G prograd as G d .
Revised: In this case, we safely set the updated gradient direction G prograd as G d .

Original: Their angle is larger than 90 ◦ (Figure 3(b)), which indicates that the few-shot downstream knowledge conflicts with general knowledge.
Revised: Their angle is larger than 90 ◦ (Figure 3(b)), which indicates that the few-shot downstream knowledge conflicts with general knowledge.

Original: In other words, optimizing the context vectors following G d will lead to the forgetting of the pre-trained general knowledge.
Revised: In other words, optimizing the context vectors following G d will lead to the forgetting of the pre-trained general knowledge.

Original: In this case, we project the G d to the orthogonal direction of G g to optimize the model for classification, which avoids increasing the KL loss.
Revised: In this case, we project the G d to the orthogonal direction of G g to optimize the model for classification, which avoids increasing the KL loss.

Original: Our ProGrad strategy is mathematically formulated as:
Revised: Our ProGrad strategy is mathematically formulated as:

Original: [Equation]
Revised: [Equation]

Original: Fig 3(c) illustrates the pipeline of our ProGrad .
Revised: Fig 3(c) illustrates the pipeline of our ProGrad .

Original: Instead of updating the context vectors using
Revised: Instead of updating the context vectors using

Original: G d in CoOp (Zhou et al., 2021), we optimize the context vectors using G prograd , which prevent the gradient direction from overfitting to few-shot downstream samples.
Revised: G d in CoOp (Zhou et al., 2021), we optimize the context vectors using G prograd , which prevent the gradient direction from overfitting to few-shot downstream samples.

Original: We further introduce λ in
Revised: We further introduce λ in

Original: Eq.
Revised: Eq.

Original: (4) to generalize the formulation, which can flexibly control the strength of general knowledge guidance in applications.
Revised: (4) to generalize the formulation, which can flexibly control the strength of general knowledge guidance in applications.

Original: In particular, λ = 1 denotes projecting G d to the orthogonal direction of
Revised: In particular, λ = 1 denotes projecting G d to the orthogonal direction of

Original: G g (Figure 3(b)), while setting λ = 0 makes ProGrad degenerate to CoOp, i.e. , CoOp is a special case of our strategy.
Revised: G g (Figure 3(b)), while setting λ = 0 makes ProGrad degenerate to CoOp, i.e. , CoOp is a special case of our strategy.

Original: We include the detailed analysis of λ in Appendix.
Revised: We include the detailed analysis of λ in Appendix.

Original: Generalization Error Analysis .
Revised: Generalization Error Analysis .

Original: We further theoretically analyze the generalization error of our
Revised: We further theoretically analyze the generalization error of our

Original: ProGrad .
Revised: ProGrad .

Original: Here, we provide a sketch proof and include the detailed justification in Appendix.
Revised: Here, we provide a sketch proof and include the detailed justification in Appendix.

Original: Our
Revised: Our

Original: ProGrad keeps the optimal value L kl of the pre-trained domain when optimizing the empirical risk on the downstream domain.
Revised: ProGrad keeps the optimal value L kl of the pre-trained domain when optimizing the empirical risk on the downstream domain.

Original: The model ˆ f prograd learned by such update rule can be viewed as optimizing the empirical risk on pre-trained and downstream domains (Yu et al., 2020):
Revised: The model ˆ f prograd learned by such update rule can be viewed as optimizing the empirical risk on pre-trained and downstream domains (Yu et al., 2020):

Original: [Equation]
Revised: [Equation]

Original: where F is the function class, and R ( · ) and ˆ R ( · ) denote the expected risk and empirical risk.
Revised: where F is the function class, and R ( · ) and ˆ R ( · ) denote the expected risk and empirical risk.

Original: We bound the generalization error of ProGrad by virtue of Rademacher Complexity (Bartlett &
Revised: We bound the generalization error of ProGrad by virtue of Rademacher Complexity (Bartlett &

Original: Mendelson, 2002) and the Theorem 6.2 in (Zhang et al., 2012).
Revised: Mendelson, 2002) and the Theorem 6.2 in (Zhang et al., 2012).

Original: The detailed proof is in Appendix.
Revised: The detailed proof is in Appendix.

Original: Theorem 1 Let X N d 1 = { x ( d ) n } N d n =1 and X N p 1 = { x ( p ) n } N p n =1 be two set of i.i.d.
Revised: Theorem 1 Let X N d 1 = { x ( d ) n } N d n =1 and X N p 1 = { x ( p ) n } N p n =1 be two set of i.i.d.

Original: samples drawn from the downstream domain D d and the pre-trained domain D p .
Revised: samples drawn from the downstream domain D d and the pre-trained domain D p .

Original: Then for any ϵ > 0 , we have with probability at least 1 − ϵ ,
Revised: Then for any ϵ > 0 , we have with probability at least 1 − ϵ ,

Original: [Equation]
Revised: [Equation]

Original: where γ F ( D, P ) is the integral probability metric (Müller, 1997) that measures the difference between the distribution of pre-trained domain and the downstream domain, R d ( F ) and R p ( F ) are the Rademacher complexity of F .
Revised: where γ F ( D, P ) is the integral probability metric (Müller, 1997) that measures the difference between the distribution of pre-trained domain and the downstream domain, R d ( F ) and R p ( F ) are the Rademacher complexity of F .

Original: Note that the bound of R ( F ) is inversely proportional to the number of training samples.
Revised: Note that the bound of R ( F ) is inversely proportional to the number of training samples.

Original: Theorem 1 shows that the generalization error R d ( ˆ f prograd ) is bounded by the empirical training risk ˆ R ( d + p ) ( ˆ f prograd ) , the two domain gap γ F ( D, P ) and the estimation error.
Revised: Theorem 1 shows that the generalization error R d ( ˆ f prograd ) is bounded by the empirical training risk ˆ R ( d + p ) ( ˆ f prograd ) , the two domain gap γ F ( D, P ) and the estimation error.

Original: The empirical training risk can be minimized to arbitrary small value when using deep models with high capacity.
Revised: The empirical training risk can be minimized to arbitrary small value when using deep models with high capacity.

Original: The estimation error that related to N p asymptotically tends to 0 as the sample size N p tends to infinity.
Revised: The estimation error that related to N p asymptotically tends to 0 as the sample size N p tends to infinity.

Original: Thanks to the large amount of pretrained samples N p , we can approximate the generalization error bound as
Revised: Thanks to the large amount of pretrained samples N p , we can approximate the generalization error bound as

Original: [Equation]
Revised: [Equation]

Original: Similarly, we have the generalization error for CoOp ˆ f coop as
Revised: Similarly, we have the generalization error for CoOp ˆ f coop as

Original: [Equation]
Revised: [Equation]

Original: Under the assumption that the gap between pre-trained and downstream domains γ ( P, D ) is small, the estimation error bound of R d ( ˆ f coop ) is at least two times greater than R d ( ˆ f prograd ) .
Revised: Under the assumption that the gap between pre-trained and downstream domains γ ( P, D ) is small, the estimation error bound of R d ( ˆ f coop ) is at least two times greater than R d ( ˆ f prograd ) .

Original: Considering that N d is typically very small in few-shot setting, our ProGrad model ˆ f prograd achieves a much lower error bound than conventional fine-tuning model like CoOp ˆ f coop .
Revised: Considering that N d is typically very small in few-shot setting, our ProGrad model ˆ f prograd achieves a much lower error bound than conventional fine-tuning model like CoOp ˆ f coop .

Original:  E XPERIMENTS 4. D ATASETS AND I MPLEMENTATION D ETAILS
Revised:  E XPERIMENTS 4. D ATASETS AND I MPLEMENTATION D ETAILS

Original: We follow CLIP, CoOp and CoCoOp to validate the effectiveness of our ProGrad on three settings: (1) few-shot classification, (2) domain generalization,and (3) base-to-new generalization.
Revised: We follow CLIP, CoOp and CoCoOp to validate the effectiveness of our ProGrad on four settings: (1) few-shot classification, (2) domain generalization, (3) base-to-new generalization, (4) cross-dataset transfer.

Revised: We report and discuss the results of cross-dataset transfer in Appendix D.

Original: Datasets.
Revised: Datasets.

Original: For evaluations of few-shot learning and base-to-new generalization, we follow CLIP and CoOp to use 11 image classification datasets, i.e. , ImageNet (Deng et al., 2009) and Caltech101 (Fei-Fei et al., 2004) for generic object classification, OxfordPets (Parkhi et al., 2012),
Revised: For evaluations of few-shot learning and base-to-new generalization, we follow CLIP and CoOp to use 11 image classification datasets, i.e. , ImageNet (Deng et al., 2009) and Caltech101 (Fei-Fei et al., 2004) for generic object classification, OxfordPets (Parkhi et al., 2012),

Original: StanfordCars (Krause et al., 2013), Flowers102 (Nilsback & Zisserman, 2008), Food101 (Bossard et al., 2014) and FGVCAircraft (Maji et al., 2013) for fine-grained image recognition, EuroSAT (Helber et al., 2019) for satellite image classification, UCF101 (Soomro et al., 2012) for action classification, DTD (Cimpoi et al., 2014) for texture classification, and SUN397 (Xiao et al., 2010) for scene recognition.
Revised: StanfordCars (Krause et al., 2013), Flowers102 (Nilsback & Zisserman, 2008), Food101 (Bossard et al., 2014) and FGVCAircraft (Maji et al., 2013) for fine-grained image recognition, EuroSAT (Helber et al., 2019) for satellite image classification, UCF101 (Soomro et al., 2012) for action classification, DTD (Cimpoi et al., 2014) for texture classification, and SUN397 (Xiao et al., 2010) for scene recognition.

Original: For domain generalization, we use ImageNet as the source dataset and select
Revised: For domain generalization, we use ImageNet as the source dataset and select

Original: ImageNetV2 (Recht et al., 2019), ImageNet-Sketch (Wang et al., 2019), ImageNet-A (Hendrycks et al., 2021b) and ImageNet-R (Hendrycks et al., 2021a) as the target datasets.
Revised: ImageNetV2 (Recht et al., 2019), ImageNet-Sketch (Wang et al., 2019), ImageNet-A (Hendrycks et al., 2021b) and ImageNet-R (Hendrycks et al., 2021a) as the target datasets.

Original: Training Details.
Revised: Training Details.

Original: For few-shot learning, following CoOp and CLIP, all models are trained with {1, 2, 4, 8, 16} shots respectively then evaluated on the full test split.
Revised: For few-shot learning, following CoOp and CLIP, all models are trained with {1, 2, 4, 8, 16} shots respectively then evaluated on the full test split.

Original: For domain generalization and base-to-new generalization, we evaluate 4-shot performance, which justifies the robustness under low-shots condition.
Revised: For domain generalization and base-to-new generalization, we evaluate 4-shot performance, which justifies the robustness under low-shots condition.

Original: All results of learning-based models are averaged over three random seeds for fair comparison.
Revised: All results of learning-based models are averaged over three random seeds.

Original: Unless otherwise stated, we adhere to CoOp to use ResNet-50 (He et al., 2016) as the backbone of image encoder.
Revised: Unless otherwise stated, we adhere to CoOp to use ResNet-50 (He et al., 2016) as the backbone of image encoder.

Original: Following CoOp and CoCoOp, the length of context tokens M is set to 16 for few-shot classification and M = 4 for base-to-new and domain generalization.
Revised: Following CoOp and CoCoOp, the length of context tokens M is set to 16 for few-shot classification and M = 4 for base-to-new, domain generalization and cross-dataset transfer.

Original: We follow the same training epochs, training schedule and the data augmentation settings in CoOp.
Revised: We follow the same training epochs, training schedule and the data augmentation settings in CoOp.

Original: λ is set toby default, except that λ is set to 0.8 for 16 shots.
Revised: λ is set to 1 by default, except that λ is set to 0.8 for 16 shots.

Original: Please refer to Appendix for more details.
Revised: Please refer to Appendix for more details.

Original: Baselines.
Revised: Baselines.

Original: We compare ProGrad with 4 baselines: (1) Zero-shot CLIP, which is based on handcrafted prompts.
Revised: We compare ProGrad with 4 baselines: (1) Zero-shot CLIP (2) Linear probe CLIP (3)

Original: We follow the prompt design in CoOp.

Original: (2) Linear probe CLIP, which trains a linear classifier on top of the CLIP image features. (3) CoOp, which learns the context prompt from data rather than hand-crafted design. (4) CoCoOp (Zhou et al., 2022), which extends CoOp by learning image conditional prompts instead of a static one.
Revised: CoOp and (4) CoCoOp.

Original: Although our method can beat some other fine-tune methods like CLIP-Adapter (Gao et al., 2021a), we mainly focus on comparing with prompt-based learning methods.
Revised: Although our method can beat some other fine-tune methods like CLIPAdapter (Gao et al., 2021a), we mainly focus on comparing with prompt-based learning methods.

Original: The results of other fine-tuning methods are in Appendix.
Revised: The results of other fine-tuning methods are in Appendix.

Original: 4. F EW -S HOT C LASSIFICATION
Revised: 4. F EW -S HOT C LASSIFICATION

Revised: Setup.

Revised: We compare with two zero-shot CLIP models, i.e. , CLIP and CLIP++ stand for using single prompt and prompt ensembling respectively (Please refer to Appendix Section C for the templates

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: Table 1 provides averaged accuracy overdatasets, where CLIP and CLIP++ stand for zero-shot results that using single prompt and prompt ensembling respectively; ProGrad and ProGrad++ stand for using single prompt and prompt ensembling as general knowledge to implement ProGrad respectively. We select the ensemble prompts from (Radford et al., 2021).
Revised: Table 1 provides averaged accuracy overdatasets.

Original: Please refer to Appendix for the details of ensemble prompts.

Original: Figure 4 illustrates the detailed comparisons over 11 datasets.
Revised: Figure 4 illustrates the detailed comparisons over 11 datasets.

Original: Overall, ProGrad achieves clear advantages over baseline models for all the few-shot settings on average performance.
Revised: Overall, ProGrad achieves clear advantages over baselines for all the few-shot settings on average performance.

Original: Specifically, ProGrad outperforms CoOp by 9 . 5% , 6 . 9% and 5 .
Revised: Specifically, ProGrad outperforms CoOp by 9 . 5% , 6 . 9% and 5 .

Original: 1% on FGVCAircraft, EuroSAT and Flowers102 given 1 shot, and the average improvement over 11 datasets is 3 .2% .
Revised: 1% on FGVCAircraft, EuroSAT and Flowers102 given 1 shot, and the average improvement over 11 datasets is 3 .

Original: These results demonstrate the anti-overfitting ability of our ProGrad when the training samples are extremely limited.
Revised: These results demonstrate the anti-overfitting ability of our ProGrad when the training samples are extremely limited.

Original: Furthermore, leveraging prompt ensembling can further explore the potential of ProGrad .
Revised: Furthermore, leveraging prompt ensembling can further explore the potential of ProGrad .

Original: From Table 1, with more accurate general knowledge offered by prompt ensembling, CLIP++ improves the zero-shot CLIP from 58 .
Revised: From Table 1, with more accurate general knowledge offered by prompt ensembling, CLIP++ improves the zero-shot CLIP from 58 .

Original: 77% to 59 .
Revised: 77% to 59 .

Original: 38% ; ProGrad++ increases the accuracy of ProGrad from 74 .
Revised: 38% ; ProGrad++ increases the accuracy of ProGrad from 74 .

Original: 28% to 75 .
Revised: 28% to 75 .

Original: 03% at 16 shots.
Revised: 03% at 16 shots.

Original: 4. D OMAIN G ENERALIZATION
Revised: 4. D OMAIN G ENERALIZATION

Original: This setting evaluates the generalization ability of models on a target domain which is different from the source domain.
Revised: This setting evaluates the generalization ability of models on a target domain which is different from the source domain.

Original: Conventional fine-tuning on limited data from a specific domain may mislead the model to learn spurious correlations or in-distribution patterns, resulting in a biased model with under-performance in unseen domains.
Revised: Conventional fine-tuning on limited data from a specific domain may mislead the model to learn spurious correlations or in-distribution patterns, resulting in a biased model with under-performance in unseen domains.

Original: In contrast, zero-shot CLIP does not exploit such spurious correlations or patterns, since it is not fine-tuned on that distribution.
Revised: In contrast, zero-shot CLIP does not exploit such spurious correlations or patterns, since it is not fine-tuned on that distribution.

Original: Since our ProGrad uses the general knowledge from the pre-trained domain to regularize the fine-tuning on a specific distribution, our ProGrad is hopefully robust to the distribution shift.
Revised: Since our ProGrad uses the general knowledge from the pre-trained domain to regularize the fine-tuning on a specific distribution, our ProGrad is hopefully robust to the distribution shift.

Original: As shown in Table 2, despite the exposure
Revised: As shown in Table 2, despite the exposure

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: to the source dataset, ProGrad clearly outperforms Zero-shot CLIP, CoOp and CoCoOp on all target datasets as well as the source dataset with ResNet-based and Transformer-based visual backbones.
Revised: to the source dataset, ProGrad clearly outperforms Zero-shot CLIP, CoOp and CoCoOp on all target datasets as well as the source dataset with ResNet-based and Transformer-based visual backbones.

Original: 4. B ASE - TO -N EW G ENERALIZATION
Revised: 4. B ASE - TO -N EW G ENERALIZATION

Original: (Zhou et al., 2022) claim that the static context learned by CoOp is not able to generalize to unseen classes.
Revised: [Table]

Original: We follow their setting to evaluate the generalization performance from seen classes to unseen classes.
Revised: We follow (Zhou et al., 2022) to evaluate the generalization performance from seen classes to unseen classes.

Original: All the classes are equally divided into two groups, i.e. , base classes and new classes, and all the methods are only trained on base classes and tested on both base classes and novel classes.
Revised: All the classes are equally divided into two groups, i.e. , base classes and new classes, and all the methods are only trained on base classes and tested on both base classes and novel classes.

Original: The harmonic mean of base-class and new-class accuracies is reported to evaluate the trade-off.
Revised: The harmonic mean of base-class and new-class accuracies is reported to evaluate the trade-off.

Original: [Table]

Original: Compared to CoOp and CoCoOp, ProGrad also generalizes well to the new classes.
Revised: Compared to CoOp and CoCoOp, ProGrad also generalizes well to the new classes.

Original: From Table 3, we observed that ProGrad achieves the best average performance in terms of all metrics.
Revised: From Table 3, we observed that ProGrad achieves the best average performance in terms of all metrics.

Original: In contrast, CoOp and CoCoOp fail in new classes: their performance is consistently worse than zero-shot CLIP.
Revised: In contrast, CoOp and CoCoOp fail in new classes: their performance is consistently worse than zero-shot CLIP.

Original: These results highlight that ProGrad has better generalizability to both seen and unseen classes.
Revised: These results highlight that ProGrad has better generalizability to both seen and unseen classes.

Original: Please refer to Appendix for the results of 11 datasets.
Revised: Please refer to Appendix for the results of 11 datasets.

Original: 4. F URTHER A NALYSIS
Revised: 4. F URTHER A NALYSIS

Original: Failure cases .
Revised: Failure cases .

Original: We further analyze the failure cases where ProGrad models predict incorrectly but
Revised: We further analyze the failure cases where ProGrad models predict incorrectly but

Original: CoOp gives right predictions.
Revised: CoOp gives right predictions.

Original: Specifically, we count the percentage of the failure cases that zero-shot
Revised: Specifically, we count the percentage of the failure cases that zero-shot

Original: CLIP models also fails in Figure 5.
Revised: CLIP models also fails in Figure 5.

Original: We found that a high proportion of the failure cases are also mis-classified by Zero-shot CLIP model (red bar in Figure 5).
Revised: We found that a high proportion of the failure cases are also mis-classified by Zero-shot CLIP model (red bar in Figure 5).

Original: This observation indicates that the general direction G g generated by imprecise zero-shot general knowledge is detrimental to model generalization.
Revised: This observation indicates that the general direction G g generated by imprecise zero-shot general knowledge is detrimental to model generalization.

Original: As the number of samples increases, the downstream knowledge represented by G d becomes more accurate and unbiased.
Revised: As the number of samples increases, the downstream knowledge represented by G d becomes more accurate and unbiased.

Original: As expected, we observe that the red bar becomes larger.
Revised: As expected, we observe that the red bar becomes larger.

Original: Conflict of knowledge.
Revised: Conflict of knowledge.

Original: ProGrad requires the updated gradient direction be acute to the general knowledge gradient directions.
Revised: ProGrad requires the updated gradient direction be acute to the general knowledge gradient directions.

Original: We explore how this constraint helps to defuse the conflicts of domain-
Revised: We explore how this constraint helps to defuse the conflicts of domain-

Original: [Figure]
Revised: [Figure]

Original: specific and general knowledge by visualizing the angle between their representative gradients during training (angle between G d and G g ).
Revised: specific and general knowledge by visualizing the angle between their representative gradients during training (angle between G d and G g ).

Original: As depicted in Figure 6, for the training without G prograd , the angle between G d and G g converges to 90 degree due to the fact that “all high-dimensional random vectors are almost always orthogonal to each other” (Cai et al., 2013).
Revised: As depicted in Figure 6, for the training without G prograd , the angle between G d and G g converges to 90 degree due to the fact that “all high-dimensional random vectors are almost always orthogonal to each other” (Cai et al., 2013).

Original: Intuitively, without any constraint, the optimization direction G d is independent to the general direction, and the average angle would be around 90 degree ( i.e. , orthogonal).
Revised: Intuitively, without any constraint, the optimization direction G d is independent to the general direction [Table] and the average angle would be around 90 degree ( i.e. , orthogonal).

Original: In contrast, utilizing G prograd during training leads to the result that the angle finally converge to an obtuse angle.
Revised: In contrast, utilizing G prograd during training leads to the result that the angle finally converge to an obtuse angle.

Original: The reason is that G prograd intervenes the model to learn the downstream knowledge aligned with the general knowledge and leads to the insufficient learning of downstream knowledge that is incompatible with the general knowledge.
Revised: The reason is that G prograd intervenes the model to learn the downstream knowledge aligned with the general knowledge and leads to the insufficient learning of downstream knowledge that is incompatible with the general knowledge.

Original: As training stabilizes, G d struggles to learn the conflicting knowledge, reflecting an obtuse angle to the
Revised: As training stabilizes, G d struggles to learn the conflicting knowledge, reflecting an obtuse angle to the

Original: G g .
Revised: G g .

Original: Thanks to ProGrad , we discard such conflicting knowledge to avoid forgetting.
Revised: Thanks to ProGrad , we discard such conflicting knowledge to avoid forgetting.

Original: [Table]
Revised: [Table]

Original: Average accuracy (

Original: [Table]
Revised: [Table]

Original: over 11 datasets

Original: [Table]
Revised: [Table]

Original: Comparison with conventional knowledge distillation.
Revised: Comparison with conventional knowledge distillation.

Original: Since our ProGrad utilizes the gradient direction of knowledge distillation loss as regularization, one may wonder whether our
Revised: Since our ProGrad utilizes the gradient direction of knowledge distillation loss as regularization, one may wonder whether our

Original: ProGrad is indeed conventional knowledge distillation.
Revised: ProGrad is indeed conventional knowledge distillation.

Original: We answer this question by investigating whether a simple knowledge distillation ( i.e. ,
Revised: We answer this question by investigating whether a simple knowledge distillation ( i.e. ,

Original: L total =
Revised: L total =

Original: L ce + α · L kd ) can achieve similar performance as our ProGrad .
Revised: L ce + α · L kd ) can achieve similar performance as our ProGrad .

Original: We repeated the fewshot experiments on 11 datasets with a variety of α and report the average results in Table 4.
Revised: We repeated the fewshot experiments on 11 datasets with a variety of α and report the average results in Table 4.

Original: Overall,
Revised: Overall,

Original: ProGrad outperforms KD for various few-shot settings.
Revised: ProGrad outperforms KD for various few-shot settings.

Original: Although KD with small α ≤ 1 promotes
Revised: Although KD with small α ≤ 1 promotes

Original: CoOp in low-shot ( e.g. , 1, 2 and 4 shots), the performance drops when number of shots is large (see 8 and 16 shots).
Revised: CoOp in low-shot ( e.g. , 1, 2 and 4 shots), the performance drops when number of shots is large (see 8 and 16 shots).

Original: These results indicate that our
Revised: These results indicate that our

Original: ProGrad works differently from KD and is more robust to the number of training samples.
Revised: ProGrad works differently from KD and is more robust to the number of training samples.

Original: Applying ProGrad to conventional fine-tune paradigm.
Revised: Applying ProGrad to conventional fine-tune paradigm.

Original: We are also interested in whether
Revised: We are also interested in whether

Original: ProGrad can be applied to the conventional “pre-train then fine-tune” paradigm.
Revised: ProGrad can be applied to the conventional “pre-train then fine-tune” paradigm.

Original: Specifically, we plug in an additional cosine classifier on top of the visual backbone and compare the performance by conducting the few-shot experiments.
Revised: Specifically, we plug in an additional cosine classifier on top of the visual backbone and compare the performance by conducting the few-shot experiments.

Original: Table 5 shows that conventional fine-tuning can benefit from our ProGrad .
Revised: Table 5 shows that conventional fine-tuning can benefit from our ProGrad .

Original: The implementation details and the result of each dataset are provided in Appendix.
Revised: The implementation details and the result of each dataset are provided in Appendix.

Revised: We also analyze the upper bound performance of ProGrad and effect of the hyper-parameter λ .

Revised: The results and discussion are presented in Appendix D.3 and Appendix D.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we pointed out the over-fitting issues of existing prompt tuning methods for few-shot generalization, which heavily relies on early stopping and data augmentation to promote zeroshot inference.
Revised: In this paper, we pointed out the over-fitting issues of existing prompt tuning methods for few-shot generalization, which heavily relies on early stopping and data augmentation to promote zeroshot inference.

Original: We proposed a prompt tuning method ProGrad that regularize each tuning step not to conflict with the general knowledge of the hand-crafted prompt.
Revised: We proposed a prompt tuning method ProGrad that regularize each tuning step not to conflict with the general knowledge of the hand-crafted prompt.

Original: Experiments on few-shot classification, base-to-new generalization and domain generalization over 11 datasets demonstrate the effectiveness and efficiency of our ProGrad .
Revised: Experiments on few-shot classification, base-to-new generalization and domain generalization over 11 datasets demonstrate the effectiveness and efficiency of our ProGrad .

Original: In the future, we will explore how to apply ProGrad on other tasks like object detection and segmentation.
Revised: In the future, we will explore how to apply ProGrad on other tasks like object detection and segmentation.

Original: E THIC S TATEMENT
Revised: E THIC S TATEMENT

Original: No human subjects were involved during the research and developments of this work.
Revised: No human subjects were involved during the research and developments of this work.

Original: All of our experiments were conducted on the standard benchmarks in the lab-based, controlled environment.
Revised: All of our experiments were conducted on the standard benchmarks in the lab-based, controlled environment.

Original: Thus, due to the abstract nature of this work, it has minimal concerns regarding issues such as discrimination/bias/fairness, privacy, etc.
Revised: Thus, due to the abstract nature of this work, it has minimal concerns regarding issues such as discrimination/bias/fairness, privacy, etc.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: In this paper we conduct the experiments three times and report the mean and standard deviation (confident interval at 95%) values to alleviate the randomness of the starting seed.
Revised: In this paper we conduct the experiments three times and report the mean and standard deviation (confident interval at 95%) values to alleviate the randomness of the starting seed.

Original: In Appendix
Revised: In Appendix

Original: Section D.2, we provide the full details of our experimental settings.
Revised: Section D.4, we provide the full details of our experimental settings

Original: A J USTIFICATION FROM G ENERALIZATION E RROR
Revised: A J USTIFICATION FROM G ENERALIZATION E RROR

Original: We further analyze the generalization error bound of our ProGrad .
Revised: We further analyze the generalization error bound of our ProGrad .

Original: We define the expected risk R ( · ) and empirical risk ˆ R ( · ) of a classifier f on domain D as
Revised: We define the expected risk R ( · ) and empirical risk ˆ R ( · ) of a classifier f on domain D as

Original: [Equation]
Revised: [Equation]

Original: where ℓ ( f ( X ) , Y ) denotes the cross-entropy and N is the volume of training data.
Revised: where ℓ ( f ( X ) , Y ) denotes the cross-entropy and N is the volume of training data.

Original: We are interested in the downstream domain D d and pre-trained domain D p , respectively.
Revised: We are interested in the downstream domain D d and pre-trained domain D p , respectively.

Original: Let F be a function class, the conventional fine-tune model ˆ f coop is trained on D d by
Revised: Let F be a function class, the conventional fine-tune model ˆ f coop is trained on D d by

Original: [Equation]
Revised: [Equation]

Original: The zero-shot CLIP model ˆ f p is considered to be trained on D p by
Revised: The zero-shot CLIP model ˆ f p is considered to be trained on D p by

Original: [Equation]
Revised: [Equation]

Original: For the implementation of ProGrad , we initialize the model ˆ f prograd using the pre-trained model ˆ f p .
Revised: For the implementation of ProGrad , we initialize the model ˆ f prograd using the pre-trained model ˆ f p .

Original: We regularize each training step not to increase the KL divergence between the predictions of ˆ f prograd and ˆ f p .
Revised: We regularize each training step not to increase the KL divergence between the predictions of ˆ f prograd and ˆ f p .

Original: In this way, ˆ f prograd can keep the optimal value of the pre-trained domain L kl when optimizing the empirical risk on the downstream domain.
Revised: In this way, ˆ f prograd can keep the optimal value of the pre-trained domain L kl when optimizing the empirical risk on the downstream domain.

Original: The model ˆ f prograd learned by our ProGrad can be viewed as optimizing the empirical risk on both domains:
Revised: The model ˆ f prograd learned by our ProGrad can be viewed as optimizing the empirical risk on both domains:

Original: [Equation]
Revised: [Equation]

Original: Based on Theorem 4.1 of (Yang et al., 2021), assuming that the neural network has L layers with parameters matrices W 1 , ..., W L , and their Frobenius norm are at most M 1 , ..., M L and the activation functions are 1-Lispschitz continuous, positive-homogeneous, and applied element-wise.
Revised: Based on Theorem 4.1 of (Yang et al., 2021), assuming that the neural network has L layers with parameters matrices W 1 , ..., W L , and their Frobenius norm are at most M 1 , ..., M L and the activation functions are 1-Lispschitz continuous, positive-homogeneous, and applied element-wise.

Original: The output of the neural network is the softmax function that predicts c classes.
Revised: The output of the neural network is the softmax function that predicts c classes.

Original: Let F be a function class with the range [ a, b ] .
Revised: Let F be a function class with the range [ a, b ] .

Original: Distribution is such that ∥ x ∥ ≤ B .
Revised: Distribution is such that ∥ x ∥ ≤ B .

Original: Let X N d 1 = { x ( d ) n } N d n =1 and X N p 1 = { x ( p ) n } N p n = be two set of i.i.d.
Revised: Let X N d 1 = { x ( d ) n } N d n =1 and X N p 1 = { x ( p ) n } N p n = be two set of i.i.d.

Original: samples drawn from the downstream domain D d and the pre-trained domain D p .
Revised: samples drawn from the downstream domain D d and the pre-trained domain D p .

Original: Then for any ϵ > 0 , we have with probability at least 1 − ϵ ,
Revised: Then for any ϵ > 0 , we have with probability at least 1 − ϵ ,

Original: [Equation]
Revised: [Equation]

Original: where γ F ( D, P ) is the integral probability metric (Müller, 1997) that measures the difference between the distribution of pre-trained domain and the downstream domain.
Revised: where γ F ( D, P ) is the integral probability metric (Müller, 1997) that measures the difference between the distribution of pre-trained domain and the downstream domain.

Original: The Eq.
Revised: The Eq.

Original: (13) shows that the generalization error R d ( ˆ f prograd ) is bounded by the empirical training risk ˆ R ( d + p ) ( ˆ f prograd ) , the two domain gap γ F ( D, P ) and the estimation error that is inversely proportional to number of training samples, i.e. , N d and N p .
Revised: (13) shows that the generalization error R d ( ˆ f prograd ) is bounded by the empirical training risk ˆ R ( d + p ) ( ˆ f prograd ) , the two domain gap γ F ( D, P ) and the estimation error that is inversely proportional to number of training samples, i.e. , N d and N p .

Original: The empirical training risk can be minimized to arbitrary small value and the estimation error that related to N p asymptotically tends to 0 as the sample size N p tends to infinity.
Revised: The empirical training risk can be minimized to arbitrary small value and the estimation error that related to N p asymptotically tends to 0 as the sample size N p tends to infinity.

Original: Thanks to the large amount of pretrained samples N p , we can approximate the generalization error bound for the model learned by ProGrad as
Revised: Thanks to the large amount of pretrained samples N p , we can approximate the generalization error bound for the model learned by ProGrad as

Original: [Equation]
Revised: [Equation]

Original: Similarly, we have the generalization error for ˆ f coop as
Revised: Similarly, we have the generalization error for ˆ f coop as

Original: R d ( ˆ f coop )
Revised: R d ( ˆ f coop )

Original: [Equation]
Revised: [Equation]

Original: If the gap between the pre-trained domain D p and the downstream domain D d is very small, the γ F ( D, P ) will tend to 0.
Revised: If the gap between the pre-trained domain D p and the downstream domain D d is very small, the γ F ( D, P ) will tend to 0.

Original: Under this assumption, the estimation error bound of R d ( ˆ f coop ) is at least 2 times greater than R d ( ˆ f prograd ) .
Revised: Under this assumption, the estimation error bound of R d ( ˆ f coop ) is at least 2 times greater than R d ( ˆ f prograd ) .

Original: Considering that in few-shot setting, N d is typically very small, which makes our ProGrad model ˆ f prograd a much lower error bound than conventional fine-tuning model ˆ f coop .
Revised: Considering that in few-shot setting, N d is typically very small, which makes our ProGrad model ˆ f prograd a much lower error bound than conventional fine-tuning model ˆ f coop .

Original: B A DDITIONAL I MPLEMENTATION D ETAILS
Revised: B A DDITIONAL I MPLEMENTATION D ETAILS

Original: For ProGrad implementation, we first initialize the learnable context vector v with the word embeddings of the zero-shot hand-crafted prompt.
Revised: For ProGrad implementation, we first initialize the learnable context vector v with the word embeddings of the zero-shot hand-crafted prompt.

Original: Concretely, if the context length M is 16 and the hand-crafted prompt is “ a photo of a ”, which only has 4 tokens, we initialize the formercontext vectors with zeros and the last 4 context vectors with the word embedding of “ a photo of a ”.
Revised: Concretely, if the context length M is 16 and the hand-crafted prompt is “ a photo of a ”, which only has 4 tokens, we initialize the formercontext vectors with zeros and the last 4 context vectors with the word embedding of “ a photo of a ”.

Original: We follow the training settings of CoOp (Zhou et al., 2021): All prompt-based models are trained by SGD with an initial learning rate of 0.002 which is decayed by the cosine annealing rule.
Revised: We follow the training settings of CoOp (Zhou et al., 2021): All prompt-based models are trained by SGD with an initial learning rate of 0.002 which is decayed by the cosine annealing rule.

Original: During the first epoch, we use the warm-up trick by fixing the learning rate to 1 × 10 − 5 to alleviate the gradient explosion.
Revised: During the first epoch, we use the warm-up trick by fixing the learning rate to 1 × 10 − 5 to alleviate the gradient explosion.

Original: The training epoch is set to 50 for all shots of experiments of ImageNet dataset.
Revised: The training epoch is set to 50 for all shots of experiments of ImageNet dataset.

Original: For the rest 10 datasets, the training epoch is set to 50 for 1 shot, 100 for 2/4 shots and 200 for 8/ shots.
Revised: For the rest 10 datasets, the training epoch is set to 50 for 1 shot, 100 for 2/4 shots and 200 for 8/ shots.

Original: We train all prompt-based model with batch size of 32 expect for CoCoOp.
Revised: We train all prompt-based model with batch size of 32 expect for CoCoOp.

Original: As described in (Zhou et al., 2022), CoCoOp consumes a significant amount of GPU memory if the batch size is set larger than one.
Revised: As described in (Zhou et al., 2022), CoCoOp consumes a significant amount of GPU memory if the batch size is set larger than one.

Original: We set the batch size to 1, following their original setting.
Revised: We set the batch size to 1, following their original setting.

Original: Our experiments are conducted on one 2080Ti GPU for all datasets except ImageNet where we train the models on one
Revised: Our experiments are conducted on one 2080Ti GPU for all datasets except ImageNet where we train the models on one

Original: A100 GPU.
Revised: A100 GPU.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: C H AND - CRAFTED P ROMPTS
Revised: C H AND - CRAFTED P ROMPTS

Original: The hand-crafted prompts for 11 datasets as well as the ImageNet variants are listed in Table 6.
Revised: The hand-crafted prompts for 11 datasets as well as the ImageNet variants are listed in Table 6.

Original: We select the ensemble prompts from CLIP (Radford et al., 2021), examples for ImageNet are shown in
Revised: We select the ensemble prompts from CLIP (Radford et al., 2021), examples for ImageNet are shown in

Original: Table 7.
Revised: Table 7.

Original: D A DDITIONAL E XPERIMENTS D. E FFECT OF H YPER - PARAMETER
Revised: [Table] s [Table] r d C a r s

Revised: [Table]

Revised: l

Revised: [Table]

Revised: r

Revised: [Table]

Revised: D A DDITIONAL E XPERIMENTS D. C ROSS -D ATASET T RANSFER

Revised: All models are trained on ImageNet as source dataset and evaluated on the rest 10 target datasets.

Revised: The goal of this setting is to demonstrate the potential to transfer beyond a single dataset.

Revised: The results are presented in Table 8.

Revised: As shown, our ProGrad not only achieves the highest performance on source datasets but also outperforms other baselines 9 out of 10 datasets

Revised: [Table]

Revised: D. U PPER B OUND OF P RO G RAD

Revised: As the regularized general gradient direction G g is the key to improve the results

Revised: [Table]

Revised: we are interested in the upper-bound of performance if we use an oracle direction G full g instead of the one offered by hand-crafted prompt.

Revised: To do so

Revised: [Table]

Revised: we first optimize a prompt with plain cross-entropy loss on the full dataset to create G full g

Revised: and then use such gradient to implement ProGrad.

Revised: The results are shown in Table 9.

Revised: The results indicate a more accurate regularization direction G full g can elicit a stronger

Revised: ProGrad model.

Revised: [Table]

Revised: D. E FFECT OF H YPER - PARAMETER

Original: We further analyze the effect of the hyper-parameter λ described in Eq.
Revised: We further analyze the effect of the hyper-parameter λ described in Eq.

Original: (4) in the main paper.
Revised: (4) in the main paper.

Original: Results are shown in Table.
Revised: Results are shown in Table.

Original: As discussed in Section 3.2 in the main paper, a smaller λ weakens the general knowledge regularization, which results in a inferior performance under low-shot setting for most datasets.
Revised: As discussed in Section 3.2 in the main paper, a smaller λ weakens the general knowledge regularization, which results in a inferior performance under low-shot setting for most datasets.

Original: However, for DTD in Table 8, using a smaller λ = 0 . 9 to reduce the general knowledge regularization can improve the 16 shots results.
Revised: However, for DTD in Table 10, using a smaller λ = 0 . 9 to reduce the general knowledge regularization can improve the 16 shots results.

Original: One possible reason is that texture images of DTD has large gap with the CLIP pre-trained images that collected from the Internet, stronger regularization from pre-trained knowledge might be detrimental to the fine-tune performance if downstream data is sufficient.
Revised: One possible reason is that texture images of DTD has large gap with the CLIP pre-trained images that collected from the Internet, stronger regularization from pre-trained knowledge might be detrimental to the fine-tune performance if downstream data is sufficient.

Original: [Table]
Revised: [Table]

Original: D. A DDITIONAL F EW - SHOT C LASSIFICATION R ESULTS
Revised: A DDITIONAL F EW - SHOT C LASSIFICATION R ESULTS

Original: In this section, we further provide the detailed few-shot classification results of other learning-based fine-tuning methods with confidence interval at 95% in Table 9 and Table 10.
Revised: In this section [Table] we further provide the detailed few-shot classification results of other learning-based fine-tuning methods with confidence interval at 95% in Table 11 and Table 12.

Original: Cosine.
Revised: Cosine.

Original: As described in Section 4.5 of the main paper, we plug in an additional cosine classifier on top of the visual backbone and trained on downstream dataset.
Revised: As described in Section 4.5 of the main paper, we plug in an additional cosine classifier on top of the visual backbone and trained on downstream dataset.

Original: CoOp learns the context prompt from data rather than hand-crafted design.
Revised: CoOp learns the context prompt from data rather than hand-crafted design.

Original: CLIP-Adapter learns additional feature adapter to boost conventional fine-tuning results.
Revised: CLIP-Adapter learns additional feature adapter to boost conventional fine-tuning results.

Original: Cosine + ProGrad employs ProGrad to the training process of cosine classifier.
Revised: Cosine + ProGrad employs ProGrad to the training process of cosine classifier.

Original: CoOp + l 2 prompt reg .
Revised: CoOp + l 2 prompt reg .

Original: We further investigate whether simply using the l 2 distance between learned prompt vector v and the word embedding vector of hand-crafted prompt v zs as the regularization can improve few-shot performance, i.e. , L total ( v ) =
Revised: We further investigate whether simply using the l 2 distance between learned prompt vector v and the word embedding vector of hand-crafted prompt v zs as the regularization can improve few-shot performance, i.e. , L total ( v ) =

Original: L ce ( v ) + α ∥ v − v zs ∥ 2 , where we select α = 0 . 01 .
Revised: L ce ( v ) + α ∥ v − v zs ∥ 2 , where we select α = 0 . 01 .

Original: CoOp + GM applies gradient matching method (Yu et al., 2020) to CoOp, i.e. , we not only project the G d to the perpendicular direction of G g as the updated gradient, but also project the G g to the perpendicular direction of G d as the updated gradient to fine-tune the model alternately.
Revised: CoOp + GM applies gradient matching method (Yu et al., 2020) to CoOp, i.e. , we not only project the G d to the perpendicular direction of G g as the updated gradient, but also project the G g to the perpendicular direction of G d as the updated gradient to fine-tune the model alternately.

Original: CoOp + KD.
Revised: CoOp + KD.

Original: As described in Section 4.5 of the main paper, we apply knowledge distillation loss to
Revised: As described in Section 4.5 of the main paper, we apply knowledge distillation loss to

Original: CoOp, i.e. , L total =
Revised: CoOp, i.e. , L total =

Original: L ce + L kl
Revised: L ce + L kl

Original: CoOp + ProGrad employs ProGrad to CoOp.
Revised: CoOp + ProGrad employs ProGrad to CoOp.

Original: For all prompt-based methods, we set the context length M to 16 except for CoOp + l 2 prompt reg.
Revised: For all prompt-based methods, we set the context length M to 16 except for CoOp + l 2 prompt reg.

Original: The learned length for CoOp + l 2 prompt reg needs to be equal to the hand-crafted prompt length to compute the l 2 norm, e.g. , the M has to be 4 if the hand-crafted prompt is “ a photo of a ”.
Revised: The learned length for CoOp + l 2 prompt reg needs to be equal to the hand-crafted prompt length to compute the l 2 norm, e.g. , the M has to be 4 if the hand-crafted prompt is “ a photo of a ”.

Original: According to the average results in Table 9, we observe that our CoOp + ProGrad still achieves the best average performance.
Revised: According to the average results in Table 11, we observe that our CoOp + ProGrad still achieves the best average performance.

Original: By comparing the results of 1) Cosine and Cosine + ProGrad; and 2) CoOp and CoOp + ProGrad, we demonstrates both conventional “pre-train then fine-tune” paradigm and prompt tuning paradigm can benefit from our ProGrad .
Revised: By comparing the results of 1) Cosine and Cosine + ProGrad; and 2) CoOp and CoOp + ProGrad, we demonstrates both conventional “pre-train then fine-tune” paradigm and prompt tuning paradigm can benefit from our ProGrad .

Original: The gap between CoOp and CoOp + l 2 prompt reg demonstrates that directly regularize the learned prompt to be not far away from the hand-crafted prompt has limited improvement.
Revised: The gap between CoOp and CoOp + l 2 prompt reg demonstrates that directly regularize the learned prompt to be not far away from the hand-crafted prompt has limited improvement.

Original: By digging into CoOp + KD and
Revised: By digging into CoOp + KD and

Original: CoOp + GM, we find performance improvement by introducing the general knowledge.
Revised: CoOp + GM, we find performance improvement by introducing the general knowledge.

Original: However, their performance still under-performs our CoOp + ProGrad.
Revised: However, their performance still under-performs our CoOp + ProGrad.

Original: This is because 1) CoOp + KD learns the average knowledge from two domains which still allows the fine-tuned model to learn from the downstream knowledge that conflicts with the general knowledge; 2) CoOp + MD additional requires the fine-tuned model to discards the general knowledge that is not aligned with the downstream knowledge, as the downstream data is limited, the inaccurate estimation of G d will lead the model focus on biased general knowledge.
Revised: This is because 1) CoOp + KD learns the average knowledge from two domains which still allows the fine-tuned model to learn from the downstream knowledge that conflicts with the general knowledge; 2) CoOp + MD additional requires the fine-tuned model to discards the general knowledge that is not aligned with the downstream knowledge, as the downstream data is limited, the inaccurate estimation of G d will lead the model focus on biased general knowledge.

Original: D. A DDITIONAL R ESULTS FOR B ASE - TO -N EW G ENERALIZATION
Revised: D. A DDITIONAL R ESULTS FOR B ASE - TO -N EW G ENERALIZATION

Original: Table 11 further presents the results for base-to-new generalization on each of the 11 datasets.
Revised: Table 13 further presents the results for base-to-new generalization on each of the 11 datasets.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 77.28 ± 0.
Revised: 77.28 ± 0.

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]