Paper: MetaCoCo: A New Few-Shot Classification Benchmark with Spurious Correlation (ICLR-2024)

Reviewer comment (Writing):
Contributions are not clearly and accurately stated.

Author response:
We thank the reviewer for pointing out this issue. **We have added a concise and clear contribution summary of our paper in the “Introduction” section.** In a nutshell, our main contributions can be summarized as follows:

-	We propose a large-scale few-shot classification benchmark with spurious-correlation shifts arising from various contexts in real-world scenarios, named MetaCoCo, in which each class (or concept) consists of different backgrounds (or contexts).

-	We further propose a metric by using a pre-trained vision-language model to quantify
and compare the extent of spurious correlations on MetaCoCo and other benchmarks.

-	We conduct extensive experiments on MetaCoCo to evaluate and compare most recent methods in few-shot classification, domain shifts, overfitting, and self-supervised learning.

Reviewer comment (Writing):
The explanation for the spurious-correlation few-shot classification is not easy to understand.

Author response:
Thank you for the comment. In this paper, we follow the existing literature addressing out-of-distribution problems in computer vision [1, 2, 3] to adopt the terminologies “conceptual information” and “contextual information”. Specifically, **the conceptual information in an image refers to the object of the class,** while **the contextual information in an image refers to the background or the environment.** For example, consider an image with a dog in green grass, and the true class label is "dog". Then "dog's nose" can be regarded as a conceptual information, while "green grass" can be regarded as a contextual information. As shown in Figure 1 (b), the conceptual information includes “dog”, “bicycle”, “train”, “cat”, etc., whereas the contextual information includes “autumn”, “snow”, “rock”, “water”, etc.

Reviewer comment (Experiments):
In the experiment, why not use the most recent methods for comparison? I have only listed some references for 2021-2022, and the authors can use other recent methods rather than the enumerated ones. I'm just curious about how recent methods perform on the MetaCoCo dataset.

Author response:
We continue with our response to W2 to further explain the spurious-correlation in few-shot classification. For the aforementioned example, that is, an image with a dog in green grass, a well-trained classifier may capture both conceptual and contextual information: "dog's nose" and "green grass". However, given that the true label of the image is “dog”, we know that **the classifier produces a spurious correlation between the contextual information "green grass" and the image label “dog”.**

Different from the widely known Meta-dataset with cross-domain shifts shown in Figure 1 (a), the proposed MetaCoCo as shown in Figure 1 (b) further emphasizes the existence of spurious-correlations in the real-world scenarios. For example, in base classes we have an image "dog in green grass" (and the true class label is "dog"), while in novel classes we instead have an image "cat in green grass" (and the true class label is "cat"). **Thus, we need the classifier to be able to distinguish between these conceptual and contextual information.**

Reviewer comment (Writing):
How to understand the conceptual and contextual information mentioned in the manuscript?

Author response:
We thank the useful suggestions from the reviewer. **We added more experiments to compare the performance of the 8 most recent few-shot classification methods on the proposed MetaCoCo.** The experimental results are shown as below.

| Method | Conference | Backbone | Type | GL | LL | TT | $1$-shot | $5$-shot |
|:---------| :---------: | :---------: | :---------: | :---------: | :---------: | :---------: | :---------: | :---------: |
| CDKT+ML [4] | NeurIPS 2023 | ResNet18 | Meta |  | &check; | &check; | 44.86 | 61.42  |
| CDKT+PL [4] | NeurIPS 2023 | ResNet18 | Meta |  | &check; | &check; |43.21 | 59.87 |
| DeepBDC [5] | CVPR 2022 | ResNet12 | Metric |  | &check; | &check; |46.78 | 62.54 |
| TSA+DETA [6] | ICCV 2023 | ResNet18 | Metric | | &check; | &check; | 51.42 | 61.58 | 
| PUTM [7] | ICCV 2023 | ResNet18 | Metric | | &check; | &check; | 60.23 | 72.36 |
| FRN [8] | CVPR 2021 | ResNet12 | Fine-tuning | &check; | | &check; | 50.23 | 60.56 |
| Yang et al [9] | ECCV 2022 | ResNet12 | Fine-tuning | &check; | | &check; | 58.01 | 69.32 |
| LP-FT-FB [10] | ICLR 2023 | ResNet12 | Fine-tuning | &check; | &check; | &check; | 56.21 | 70.21 |
| | | | | | | | | | 

From the table, we find that: (1) The performance of these methods is significantly reduced on our proposed MetaCoCo dataset. (2) The performance of fine-tuning based methods is better than other methods, such as LP-FT-FB [10]. 

**In addition to the methods mentioned by the reviewer, we also verify diverse methods including vision-language pre-trained methods.**

| Method | Conference | Backbone | Type | Base (1-shot) | New (1-shot) | HM (1-shot) | Base (16-shot) | New (16-shot) | HM (16-shot) |
|:---------| :---------: | :---------: | :---------: | :---------: | :---------: | :---------: | :---------: | :---------: | :---------: |
| CLIP [11] | ICML 2021 | ResNet50 | Zero-shot CLIP | 68.12 | 72.19 | 70.10 | 68.12 | 72.19 | 70.10 |
| CoOp [12] | IJCV 2022 | ResNet50 | Prompt-tuning CLIP | 61.22 | 63.62 | 62.40 | 47.55 | 40.90 | 43.98 |
| CoCoOp [13] | CVPR 2022 | ResNet50 | Prompt-tuning CLIP | 59.86 | 63.88 | 61.80 | 36.88 | 46.10 | 40.98 |
| KgCoOp [14] | CVPR 2023 | ResNet50 | Prompt-tuning CLIP | 67.95 | 72.05 | 69.94 | 49.18 | 48.78 | 48.98 |
| LFA [15] | ICCV 2023 | ResNet50 | Prompt-tuning CLIP | 66.87 | 73.01 | 69.81 | 49.23 | 48.12 | 48.67 |
| ProGrad [16] | ICCV 2023 | ResNet50 |Prompt-tuning CLIP | 65.23 | 69.39 | 67.25 | 49.14 | 47.36 | 48.23 |
| | | | | | | | | | | 
| CLIP [11] | ICML 2021 | ViT-B/16 | Zero-shot CLIP | 73.19 | 75.29 | 74.23 | 73.19 | 75.29 | 74.23  |
| CoOp [12] | IJCV 2022 | ViT-B/16 | Prompt-tuning CLIP | 64.95 | 68.87 | 66.85 |  49.98 | 45.40 | 47.58 |
| CoCoOp [13] | CVPR 2022 | ViT-B/16 | Prompt-tuning CLIP | 65.60 | 69.50 | 67.49 | 49.35 | 47.84 | 48.58 |
| KgCoOp [14] | CVPR 2023 | ViT-B/16 | Prompt-tuning CLIP | 71.22 | 74.74 | 72.94 | 51.21 | 50.99 | 51.10 |
| LFA [15] | ICCV 2023 | ViT-B/16 | Prompt-tuning CLIP | 71.56 | 74.98 | 73.23 | 52.04 | 49.65 | 50.82 |
ProGrad [16] | ICCV 2023 | ViT-B/16 | Prompt-tuning CLIP | 70.81 | 73.27 | 72.02 | 51.48 | 48.02 | 49.69 |
| MaPLe [17] | CVPR 2023 | ViT-B/16 | Prompt-tuning CLIP | 60.31 | 68.94 | 64.34 | 48.04 | 49.47 | 48.74 |
| | | | | | | | | | | 

The above table shows the experimental results of VLMs on MetaCoCo, where base and new represent the accuracy of the base and new classes, respectively. HM is the harmonic mean of accuracy on base and new data. From the above table, we find two interesting phenomenons: (1) The more samples are used in the model prompt-tuning phase, the worse the performance of the model will be. For example, the performance drops from 62.40% with 1-shot to 43.98% with 16-shot in CoOp with ResNet50. This further indicates that the more samples used, the greater the shifts in spurious correlations introduced, resulting in reduced model performance. (2) The performance of prompt-tuning methods is lower than zero-shot CLIP in all settings. The possible reason is that prompt-tuning methods overfit to spurious information, therefore, affecting the generalization ability of the model.

Reviewer comment (Experiments):
The network structure of ResNet-50 is not explained in EXPERIMENTAL SETUP; it is suggested that the author explain it in backbone architectures.

Author response:
We thank the useful suggestions from the reviewer. For **network structure,** we follow [18] to implement ResNet50 using 50 layers, including 48 convolutional layers and 2 fully connected layers, with the detailed network structure summarized in below.

| Layer name | output size | ResNet50 |
|:---------| :---------: | :---------: |
| conv1 | 112x112 | 7x7, 64, stride 2 |
| conv2_x | 56x56 | 3x3 max pool, stride 2 |
| conv2_x | 56x56 | 1x1, 64 |
| conv2_x | 56x56 | 3x3, 64 |
| conv2_x | 56x56 | 1x1, 256 |
| conv3_x | 28x28 | 1x1, 128 |
| conv3_x | 28x28 | 3x3, 128 |
| conv3_x | 28x28 | 1x1, 512 |
| conv4_x | 14x14 | 1x1, 256 |
| conv4_x | 14x14 | 3x3, 256 |
| conv4_x | 14x14 | 1x1, 1024 |
| conv5_x | 7x7 | 1x1, 512 |
| conv5_x | 7x7 | 3x3, 512 |
| conv5_x | 7x7 | 1x1, 2048 |
| | 1x1 | average pool, 1000-d fc, softmax |
| | | | 

Specifically, each layer in conv2_x and conv5_x are repeated with 3 times, each layer in conv3_x are repeated with 4 times, and each layer in conv4_x are repeated with 6 times. The building block of ResNet50 is the residual block, which typically contains two convolutional layers along with the skip connection. The network architecture includes global average pooling and a softmax layer for classification in the final layers.

For all experiments, we use the A100 80G GPU as the computational resource. We tune batch size in $\{128, 256, 512\}$. For learning rate, optimizer, and weight decay, **we follow [20,21] as a comprehensive library for few-shot learning to implement using fixed parameter combinations as summarized in below.**

| Method | Conference | Backbone | Type | Learning Rate | Optimizer | Decay |
|:---------| :---------: | :---------: | :---------: | :---------: | :---------: | :---------: |
| MoCo [20] | CVPR 2020 | ResNet50 | Self-supervised learning | 0.03 | SGD | Cosine |
| SimCLR [21] | ICML 2020 | ResNet50 |  Self-supervised learning | 0.001 |  Adam | Cosine |
| | | | | | | | 

**For ease of reproducibility, we also added Table 4 in Appendix C showing the optimal parameter combinations when implementing the other 17 methods using ResNet12 as the backbone model following [19].**