Paper: In-context Prompt Learning for Test-time Vision Recognition with Frozen Vision-Language Model (ICLR-2024)

Reviewer comment (Writing):
Why use the term "in-context"? What is the "context" for an incoming test sample? What is the relationship between the context samples and the test sample?

Author response:
>Q1: Why use the term "in-context"? What is the "context" for an incoming test sample? What is the relationship between the context samples and the test sample?

A1: i) In-context learning, a new paradigm in NLP, enables large language models to infer unseen tasks by conditioning on specific in-context examples. To harness this emergent ability in large vision-language models like CLIP, we develop a method to learn adaptive visual prompts. This is achieved using just a few image-label pairs, which act as contextual information, aiding the model to swiftly adapt to new test samples. Therefore, these pairs are referred to as in-context examples.  

ii) For an incoming test sample, the "context" is in-context samples, i.e., several provided image-label pairs, to let the model do task like these in-context samples do on the test sample.  
iii) The context samples only provide the task information to do on the test sample.  
They are usually from the same target dataset, while there is no other relationships between them, e.g., category.

Reviewer comment (Experiments):
The goal of TTA is to address the domain shift on the fly. The introduction of the context samples makes the model task-specific. For each dataset in Table 2 and Table 3, do we need to construct a dataset-specific context?

Author response:
>Q2: The goal of TTA is to address the domain shift on the fly. The introduction of the context samples makes the model task-specific. For each dataset in Table 2 and Table 3, do we need to construct a dataset-specific context? If the answer is YES, I think this is not a TTA method. It is a domain/dataset-specific 1-shot or few-shot approach.

A2:  Yes. We need to construct a dataset-specific context for each dataset in Table 2 and Table 3.

However, our InCP is not a domain/dataset-specific 1-shot or few-shot approach.  
The reasons are: i) domain-specific few-shot approach is for training process, while our InCP is in the test time; ii) few-shot approach needs at least one sample from each category for training, while in-context examples in our InCP have no this constraint. We only sample five in-context examples no matter what category they belong to. iii) few-shot approaches like CoOP [1]  and CoCoOP [2] employ a shared prompt for all test samples, whereas our InCP adopts an adaptive, sample-specific prompt for each individual test sample.

Reviewer comment (Novelty):
If the answer is YES to Point 3, this is not a TTA method. It is a domain/dataset-specific 1-shot or few-shot approach.

Author response:
>Q2: The goal of TTA is to address the domain shift on the fly. The introduction of the context samples makes the model task-specific. For each dataset in Table 2 and Table 3, do we need to construct a dataset-specific context? If the answer is YES, I think this is not a TTA method. It is a domain/dataset-specific 1-shot or few-shot approach.

A2:  Yes. We need to construct a dataset-specific context for each dataset in Table 2 and Table 3.

However, our InCP is not a domain/dataset-specific 1-shot or few-shot approach.  
The reasons are: i) domain-specific few-shot approach is for training process, while our InCP is in the test time; ii) few-shot approach needs at least one sample from each category for training, while in-context examples in our InCP have no this constraint. We only sample five in-context examples no matter what category they belong to. iii) few-shot approaches like CoOP [1]  and CoCoOP [2] employ a shared prompt for all test samples, whereas our InCP adopts an adaptive, sample-specific prompt for each individual test sample.

Reviewer comment (Reproducibility):
The authors should further clarify how the context samples are selected. In Table 4, we can see the context samples are vital in the proposed method.

Author response:
>Q3: The authors should further clarify how the context samples are selected.

A3: We employ only a few number (i.e., 5) of in-context examples. Those examples are randomly chosen from a designated subset of the target dataset, termed "candidate in-context examples". Such a subset is created by selecting one image-label pair from each category, ensuring a diverse and representative sample pool.

Reviewer comment (Evaluation):
Please also provide the inference time of the proposed method to evaluate the efficiency.

Author response:
>Q4: Please also provide the inference time of the proposed method to evaluate the efficiency.

A4:  Thanks for your suggestion.  
We report the inference time of the proposed method in the following table.

| Method                 | Flower102 Infer. Time (↓) | Flower102 Top 1 acc. (↑) | Pets Infer. Time (↓) | Pets Top 1 acc. (↑) | Cars Infer. Time (↓) | Cars Top 1 acc. (↑) | Caltech101 Infer. Time (↓) | Caltech101 Top 1 acc. (↑) |
| --- | --- | --- | --- | ---| ---| ---| --- | ------------------------- |
| TPT [1] | 97.22  | 68.98     | 111.31       | 87.79    | 322.59  | 66.87    | 86.25  | 94.16     |
| InCP (Ours)            | **23.79**                 | **72.27**                | **16.31**            | **90.62**           | **69.05**            | **67.54**           | **35.54**                  | **94.69**                 |


[1] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. In IJCV, 2022.

[2] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.

Reviewer comment (Novelty):
(Major concern) This work proposed a in-context prompt learning method that used labeled samples as context during test stage. However, different from traditional in-context learning that used additional samples as a part of prompts and did not tune any parameters, these method used additional samples to tune the network rather than serve as part of prompt, which is like traditional semi-supervised learning pipeline but with fewer data and training loops. I am wondering whether the terminology of "in-context prompt learning" is proper. More evidence or reference on the terminology definition is expected during discussion.

Author response:
>Q1: I am wondering whether the terminology of "in-context prompt learning" is proper. More evidence or reference on the terminology definition is expected during discussion. 

A1:  We follow existing works [1,2,3] for this ``in-context learning'' concept, in which the model is updated using in-context examples.
Meanwhile, CLIP itself is not able to conduct in-context learning task.
To equip CLIP with this ability, our InCP introduces learnable prompt for each test sample in test-time stage.
In this way, the model can automatically understand the underlying task with in-context examples. For sample selection, in-context learning has no constraint on category in testing stage. The in-context samples can either share the same category as the current test sample or the irrelevant category. For sample quantity, in-context learning uses a small, arbitrary set of labeled samples. Therefore, our InCP belongs to the in-context learning area.

Reviewer comment (Experiments):
(Major concern) Random sampling may cause the information leakage of test set. Sec. 3.1 and Appendix A.2 indicate that the labeled data are randomly sampled for different test samples, which means that the model may see a wide range of test samples during evaluation, although they are not simultaneously seen. Compared to few-shot tuning setting, only a fixed set of labeled data is accessed. As shown in Appendix A.2, a fixed set of labeled data underperforms random sampling. I am wondering whether the performance gain is due to the information leakage caused by random sampling. A strict implementation would be using sampling from the same subset of labeled data for rather than the whole test set.

Author response:
>Q2: I am wondering whether the performance gain is due to the information leakage caused by random sampling. A strict implementation would be using sampling from the same subset of labeled data for rather than the whole test set.

A2: There is no information leakage in our InCP.
Actually, the selected 5 in-context examples do not share the same label with the test sample.
These 5 in-context examples are selected from a fixed subset of labeled data, which is composed by randomly sampling 1 sample from each category.
For each test sample, we use the same subset rather than the whole test set, which is the same setting as you suggested.

Reviewer comment (Presentation):
The figures are confusing and not well designed. First, in Figure 2, it is not clear what does token space mean. The number of input tokens are 4 and the number of output tokens are 2. Does it mean the number of input tokens equals to the number of words in the prompt, and the token net aims to compress the tokens? Second, in Figure 3, the module and tokens are denoted in many colors, but it is hard to understand what each color indicates, and what is the difference between different tokens in (c) and (d). Third, in Figure, the dimension of $P_t$ (i.e., 3) does not equal to the number of prefix tokens (i.e., 4). Is it in intention? Figure 1(c) is also confusing to me to understand what happened during the cyclic learning stage.

Author response:
>Q3: The figures are confusing and not well designed. First, in Figure 2, it is not clear what does token space mean. The number of input tokens are 4 and the number of output tokens are 2. Does it mean the number of input tokens equals to the number of words in the prompt, and the token net aims to compress the tokens? Second, in Figure 3, the module and tokens are denoted in many colors, but it is hard to understand what each color indicates, and what is the dfference between different tokens in (c) and (d). Third, in Figure, the dimension of (i.e., 3) does not equal to the number of prefix tokens (i.e., 4). Is it in intention? Figure 1(c) is also confusing to me to understand what happened during the cyclic learning stage.

A3: i) "token space" in Figure 2 refers to the embedding space of text tokens and visual tokens. The number of input tokens equals to the number of words in the prompt and the output tokens.
Here, the 4 to 2 means the dimension decrease, not the token number. Sorry for the confusion of token number.
We modify Figure 2 in the manuscript for clarification.
ii) The orange token represents a learnable visual token, the light purple tokens represent static visual tokens, the light gray tokens represent static text tokens, and the light orange tokens represent static prefix tokens in both Figure 3 (c) and (d). We update Figure 3 in the manuscript.
iii) Sorry for the confusion. The number of output tokens should be the same as the prefix tokens (i.e., 4). We modify it in the manuscript. iv) During the cyclic learning stage, we first optimize text prompt and then visual prompt in a sequential process. We update Figure 1 (c) for clarification.

Reviewer comment (Novelty):
Difference between this method and fully test-time adaptation. According to Table 1, the main difference is that this method used additional labeled data for tuning the prompts. If so, the novelty is just additional annotations in a semi-supervised manner, which is limited.

Author response:
>Q4: Difference between this method and fully test-time adaptation. According to Table 1, the main difference is that this method used additional labeled data for tuning the prompts. If so, the novelty is just additional annotations in a semi-supervised manner, which is limited.

A4: Fully test-time adaptation is a self-supervised learning method, which applies its self-supervised loss on unlabeled test sample to achieve the generalization capability of model on unseen domains. We argue that self-supervised learning, without task-specific supervision or model modifications, is insufficient for test-time adaptation, and thus propose in-context prompt learning for model adaptation. 

Semi-supervised learning typically incorporates labeled data during the training phase, amalgamating it with unlabeled data to fine-tune the model and improve its performance on unlabeled samples. Labeled data in semi-supervised learning often shares categories with the unlabeled data. In our method, there is no inherent relationship between in-context examples (labeled data) and the test sample, as they are both drawn from the same domain dataset. Our approach does not necessitate any category information about the test sample, distinguishing it from semi-supervised learning methods.

Reviewer comment (Experiments):
The ablation studies in Tables 4 and 6 are not comprehensive, which are only on three datasets compared to the main results on Table 2. Why only these three datasets are selection? How are they representative?

Author response:
>Q5: The ablation studies in Tables 4 and 6 are not comprehensive, which are only on three datasets compared to the main results on Table 2. Why only these three datasets are selection? How are they representative?

A5: We use these three datasets, i.e., Flower, DTD, Pets, since they are reported in most previous methods [4,5,6] and represent various object classification tasks.
To provide a more comprehensive evaluation, we perform additional ablation studies on other datasets, including ImageNet-R, ImageNet-S, Cars, and Caltech101. The results are detailed in the below tables. 

| Method (Top 1 acc.) | Adaptation | Imagenet-R | ImageNet-S | Cars  | Caltech |
|:--------|:------------:|:-----------------------:|:-----------------------:|:-----------------:|:-----------------------:|
| Ours w/o U-obj | Task | 76.89 | 47.24 | 65.29 | 93.96 |
| Ours w/o S-obj | Instance | 75.73 | 44.51 | 59.08 | 93.1 |
| Ours (SS-obj) | Task & Instance | **77.56** | **48.03** | **67.54** | **94.69** |


| Method (Top 1 acc.)| Imagenet-R | ImageNet-S | Cars | Caltech |
|:--------|:--------:|:--------:|:--------:|:--------:|
| Patched prompt | 70.62 | 43.69 | 65.14 | 91.68 |
| Padded prompt | 68.54 | 40.48 | 55.93 | 89.61 |
| Token prompt | 70.39 | 40.10 | 57.22 | 85.60 |
| Generic-language prompt | 76.46 | 45.29 | 64.17 | 92.01 |
| Ours (Language-aware) | **77.56** | **48.03** | **67.54** | **94.69** |

Reviewer comment (Theory):
I don’t think it is appropriate to call the proposed method an “in-context learning” method. In-context learning in both the language and vision literature does not involve optimizing any parameters including exterior ones like prompts. I think the method proposed in this work is more like few-shot learning, which updates parameters on some example images and then uses the updated parameters to run inference on the test sample.

Author response:
i) We follow existing works [1,2,3] for this "in-context learning" concept, in which the model is updated using in-context examples.  
Meanwhile, CLIP itself is not able to conduct in-context learning task. To equip CLIP with this ability, our InCP introduces learnable prompt for each test sample in test-time stage.  
In this way, the model can automatically understand the underlying task with in-context examples.  

ii) Our work is different from few-shot learning.  
The main difference lies in two factors, i.e., sample selection and quantity.  
(a) For sample selection, few-shot uses strict categories with specific number of samples, which are widely used in training stage.  
Differently, in-context learning has no constraint on category.  
The in-context samples in testing stage can either share the same category as the current test sample or the irrelevant category. It is also impractical to know the exact category of unlabeled test sample in advance.   
(b) For sample quantity, few-shot learning requires a predefined number of samples from each category, while in-context learning uses a small, arbitrary set of labeled samples-commonly just five samples. Therefore, our InCP belongs to the in-context learning area, rather than the few-shot learning.

Reviewer comment (Reproducibility):
It needs to provide more details about how Token Net is trained. If it is randomly initialized and only optimized for one step at test time, I don’t think it can “translate” text tokens into visual prompts that the vision encoder comprehends.

Author response:
Yes, the token net is randomly initialized.  
The feasibility of token net is based on CLIP, where the textual and visual encoders are already aligned well in the embedding space. The token net is initialized by text tokens, and then transferred to visual tokens with a few optimization steps on visual examples, which is easy to be converged.

Reviewer comment (Experiments):
If I understand correctly, the InCP in Section 4.1 is learned using in-context examples from the target dataset. However, CoOp and CoCoOp are learned on the ImageNet dataset. A fair comparison would be to report CoOp/CoCoOp’s results using the same few-shot examples as InCP.

Author response:
Thanks for your advice. We provide CoOp/CoCoOp’s results using the same examples as InCP in the following table.  
It shows that our InCP achieves better performance than CoOp and CoCoOp on this setting. Following your suggestion, we add this table in the manuscript for fair comparison.  

| Method (Top 1 acc.) | Flower | DTD   | Pets  | Cars  | Caltech |  
| ------ | :------: | :-----: | :-----: | :-----: | :----------: |  
| CoOP [2]  | 66.10   | 30.97 | 82.77 | 60.20  | 90.26      |  
| CoCoOP [3] | 67.23  | 31.72 | 83.14 | 59.78 | 90.43      |  
| InCP (Ours)    | **71.13** | **47.34** | **90.6**  | **67.54** | **94.69** |

Reviewer comment (Writing):
Section 3.4 can be more elaborated. Currently, it’s not very clear. For example, it seems like cyclic prompt learning involves multi-step optimization, while in the paragraph above section 3.4, it says the method is a single-step optimization.

Author response:
Thanks for your advice. We carefully update Section 3.4 in the manuscript to make it clear including the following clarification of the cyclic prompt learning.  

The paragraph above Section 3.4 is focused on visual prompt learning, which is a single-step optimization process.  
In Section 3.4, we introduce text prompts to integrate visual prompts.  
This process is a two-step optimization process, where each type of prompt is optimized in sequence, i.e., first visual then textual prompts. It forms the "cyclic prompt learning" concept.

Reviewer comment (Reproducibility):
During testing, when a new test sample comes in, do you reset $P_v$, $P_t$, and TokenNet back to its initial state? Or are the parameter updates accumulated?

Author response:
When a new test sample comes in, $P_{\text{v}}$ and $P_{\text{t}}$ is reset, while the parameters of the token net are accumulatively updated.

Reviewer comment (Reproducibility):
Is the Token net separately trained before test-time adaptation or randomly initialized? How are $P_v$ and $P_t$ initialized?

Author response:
Token net is randomly initialized at the start of test-time adaptation and accumulatively updated across the entire evaluation process.  
$P_{\text{t}}$ is initialized with prefix tokens derived from "a photo of a", which is then converted into a visual token.   
$P_{\text{v}}$ is then initialized by the above learned $P_{\text{t}}$.

Reviewer comment (Reproducibility):
How many tokens do you use for $P_v$ and $P_t$? How big is the TokenNet?

Author response:
The token numbers of $P_{\text{v}}$ and $P_{\text{t}}$ are 4. The token net consists of a single fully connected layer, with 512 and 768 feature dimension for input and output, respectively.

Reviewer comment (Reproducibility):
How many in-context examples do you use?

Author response:
We use a total of 5 in-context examples.

Reviewer comment (Experiments):
I don’t find the “generic-language prompt” baseline in Table 6.

Author response:
Sorry for the missing results. We update Table 6 in the manuscript as following.  

| Method (Top 1 acc.)| Flower | DTD | Pets |  
|---|:---:|:---:|:---:|  
| Patched prompt [4] | 57.90 | 32.51 | 77.98 |  
| Padded prompt [5] | 56.07 | 32.68 | 79.39 |  
| Token prompt | 59.32 | 34.63 | 74.19 |  
| Generic-language prompt | 63.30 | 44.74 | 85.20 |  
| Ours (Language-aware) | **71.13** | **47.34** | **90.25** |

Reviewer comment (Evaluation):
This paper requires clarification regarding InCP's performance on the SUN397 dataset (utilized in TPT) in Table 1 and the ImageNet dataset (also used in TPT) in Table 2. Providing a comprehensive comparison of InCP's performance on these specific datasets will significantly enhance the paper's clarity and the reader's understanding.

Author response:
The performance on ImageNet and SUN397 datasets is shown in the following table.
It shows that our InCP achieves best performance compared with previous zero/few-shot methods.
In particular, we only use 5 samples yet is better than few-shot methods [2,3].

| Method        | Type       | ImageNet  | SUN397    |
| ------------- | :----------: | :---------: | :---------: |
| CLIP-ViT-B/16 | Zero-shot  | 66.73     | 62.59     |
| Ensemble [1]     | Zero-shot  | 68.34     | 65.63     |
| TPT [1]          | Zero-shot  | 68.98     | 65.50      |
| CoOP [2]         | Few-shot   | 71.51     | 64.15     |
| CoCoOP [3]        | Few-shot   | 71.02     | 66.89     |
| MaPLe [4]        | Few-shot   | 70.72     | 67.01     |
| PromptSRC [5]    | Few-shot   | 71.27     | 67.10     |
| InCP (Ours)         | In-context | **71.62** | **67.93** |

Reviewer comment (Evaluation):
The performance improvement in Table 2 appears to be marginal. Further explanation or additional results may be needed to demonstrate the significance of the improvement.

Author response:
As few-shot methods, CoOP [2] and CoCoOP [3] fine-tune the prompt on ImageNet dataset using 16-shot training data per category and evaluate the generalization performance on downstream tasks. As a self-supervised method, TPT is not trained on ImageNet. It utilizes the self-supervised loss with unlabeled test samples to achieve its zero-shot generalization ability. Our InCP follows TPT, while only using a few in-context examples as domain-specific context information. Table 2 in the main paper shows that our method outperforms TPT by notable margins on two datasets (i.e., 72.27\% vs. 68.98\% on Flower dataset, and 70.26\% vs. 68.04\% on UCF101 dataset). Our InCP can also consistently outperforms CoOP and CoCoOP when using unlabeled ImageNet data (i.e., 70.26\% vs. 68.44\% on UCF101 dataset), and using the same in-context examples (i.e., 71.13\% vs. 67.23\% on Flower dataset). The following table shows the comparison with CoOp and CoCoOp using the same in-context examples.

| Method | Flower | DTD   | Pets  | Cars  | Caltech |
| ------ | :------: | :-----: | :-----: | :-----: | :----------: |
| CoOP [2]  | 66.10   | 30.97 | 82.77 | 60.20  | 90.26      |
| CoCoOP [3] | 67.23  | 31.72 | 83.14 | 59.78 | 90.43      |
| InCP (Ours)    | **71.13** | **47.34** | **90.6**  | **67.54** | **94.69** |