Paper: Zero-Shot Visual Classification with Guided Cropping (ICLR-2024)

Reviewer comment (Experiments):
ImageNetS919 and CUB are carefully selected as the benchmarks, but the number of the benchmarks are usually too limited for zero-shot classification evaluations. For more solid results, it would be nice to also report results on common image classification tasks, such as ImageNet, VTAB, and OOD benchmarks (e.g. ObjectNet) etc.

Author response:
Thank you for your suggestion. ImageNetS919 and CUB datasets are chosen in our work as these datasets are classification datasets with annotations (e.g., bounding boxes) from which object sizes can be retrieved. This allows us to perform detailed analysis under different object size conditions. These datasets are also commonly used in weakly supervised object localization task (e.g., [1]) as it needs similar annotations during evaluation.

Nevertheless, for the completeness of our paper, as suggested by the reviewer, we perform additional evaluation on more common datasets (ImageNet, ImageNetV2, Stanford Dogs). The results can be found in the appendix A.9 in the updated version of our paper. The results still demonstrate that integration of Guided Cropping can still be beneficial even in datasets with unconstrained object sizes (similar to the unconstrained variants of ImageNetS919 and CUB).

[1] "Weakly Supervised Object Localization as Domain Adaption", CVPR, 2022.

Reviewer comment (Evaluation):
The cost of this method was not clearly mentioned, which is meaningful given an additional detector is needed during the inference time.

Author response:
Additional parameters are required for OWL-ViT that attributes to a total of 153M model parameters. When performing a single sample inference, additional inference time required on top of CLIP is 0.15 and 0.16 seconds without and with box augmentation on a single Tesla V100-SXM2 GPU respectively. We will include these details in the final version of the paper.

Reviewer comment (Experiments):
Could we report more results from more common image classification benchmarks?

Author response:
Thank you for your suggestion. ImageNetS919 and CUB datasets are chosen in our work as these datasets are classification datasets with annotations (e.g., bounding boxes) from which object sizes can be retrieved. This allows us to perform detailed analysis under different object size conditions. These datasets are also commonly used in weakly supervised object localization task (e.g., [1]) as it needs similar annotations during evaluation.

Nevertheless, for the completeness of our paper, as suggested by the reviewer, we perform additional evaluation on more common datasets (ImageNet, ImageNetV2, Stanford Dogs). The results can be found in the appendix A.9 in the updated version of our paper. The results still demonstrate that integration of Guided Cropping can still be beneficial even in datasets with unconstrained object sizes (similar to the unconstrained variants of ImageNetS919 and CUB).

[1] "Weakly Supervised Object Localization as Domain Adaption", CVPR, 2022.

Reviewer comment (Evaluation):
Could we report the additional inference cost?

Author response:
Additional parameters are required for OWL-ViT that attributes to a total of 153M model parameters. When performing a single sample inference, additional inference time required on top of CLIP is 0.15 and 0.16 seconds without and with box augmentation on a single Tesla V100-SXM2 GPU respectively. We will include these details in the final version of the paper.

Reviewer comment (Evaluation):
In 5.4, could we also show the scores of the examples? So that we could understand a bit more possibly the confidence of the model (e.g. whether a smaller score could indicate that the model needs more context for classification?).

Author response:
Thank you for your suggestion. Visualizing the scores of these examples can indeed improve understanding of the model behaviors. We will incorporate this suggestion in the final version of our paper.

Reviewer comment (Novelty):
The observations of current limitations of CLIP and OWL-ViT models are somewhat surface-level and I believe well-known (although possibly not written down in a publication).

Author response:
In our work, we address two limitations of CLIP, its vulnerability to extraneous image regions and its sensitivity to non-semantic crops. I agree with the reviewer that some can have assumptions similar to these limitations. However, validating the assumptions and systematically quantifying impacts of these limitations are also crucial but missing from the community. For the vulnerability to extraneous image regions, we show that performance gaps between scenarios with unconstrained objects and scenarios with small objects can be up to 15-20 percents (see Figure 6 in section 5.3). The performance drops can also be observed in various supervision strategies, i.e., in zero-shot, few-shot and fully supervised models (see Table 3, 4 and 5 in the appendix). For the sensitivity to non-semantic crops, its effect is quantitatively evaluated. From Figure 4, we demonstrate that around 40% of test samples in our setup have changes in their final predictions just from small removal around image boders. These empirical studies show that impacts of these limitations are not negligible. We believe that these studies can be helpful for the research community.

Reviewer comment (Experiments):
Although the focus of the proposed approach is to correct failure cases of CLIP and OWL-ViT (i.e. small object classification), and the benchmarks were chosen to assess this specifically, it would be very useful to see the proposed method benchmarked on widely used zero-shot datasets like ImageNet.

Author response:
Thank you for your suggestion. We conducted further experiments on additional datasets (ImageNet, ImageNetV2 and Stanford Dogs). The results can be found in the appendix A.9 of the updated version of our paper. The results still demonstrate that GC-CLIP improves baseline performance even in datasets with unconstrained object sizes.

Reviewer comment (Experiments):
It would be interesting to discuss the limitations of the proposed method. For example, it focuses on image classification, but CLIP and OWL-ViT provide more than that. For example, CLIP embeddings can be used for image-text retrieval, and OWL-ViT embeddings can be used for image-conditioned detection. Can the authors' method be extended to these use cases?

Author response:
Guided Cropping can be viewed as a strategy to refine image features conditioned to classes of interest. Therefore, it is ideal to be employed in the setting of classification task since all classes of interest are known in advance. For some generic tasks (image-text retrieval, image-conditioned detection) that all classes of interest are not defined, it would not be straigtforward to employ Guided Cropping in general. 

However, on the specific scenarios that domains of interest are known in advance, Guided Cropping can also be beneficial. For example, in case of image-conditioned detection, if we already know that the domain of interest is animal, we can use Guided Cropping to refine image features using generic animal prompts (e.g., the word "animal" itself). In this case, information of unrelated contexts of query images can be discarded by Guided Cropping which could lead to better detection performance. We will include this discussion in the final version of the paper.

Reviewer comment (Evaluation):
I suggest the authors report the computational cost of GC-CLIP in the paper, including the parameters, FLOPs or the inference time, for a more comprehensive comparison with CLIP.

Author response:
**Response:** Additional parameters are required for OWL-ViT that attributes to a total of 153M model parameters. When performing a single sample inference, additional inference time required on top of CLIP is 0.15 and 0.16 seconds without and with box augmentation on a single Tesla V100-SXM2 GPU respectively. We will include these details in the final version of the paper.

Reviewer comment (Experiments):
I am confused about the necessity of combining OWL-ViT and CLIP, because the authors’ results in the experimental section show that the difference between introducing OWL-ViT for guided cropping and using random cropping is slight, more results and analysis on different datasets should be provided to illustrate the advancement of guided cropping. In particular, the authors did not report results with random cropping alone when guided cropping was used on CALIP.

Author response:
**Response:** We understand that your concern about slight improvement mainly refers to results on datasets with unconstrained object sizes (ImageNetS919 and CUB). In this case, this observation is expectable since many images of these datasets already have large enough object sizes (images have small extraneous regions) and thus do not greatly benefit from our Guided Cropping component. Note that our work is mainly motivated and designed to address the challenging cases of input images with small object sizes and large extraneous regions. Henceforth, our improvements are more evident on ImageNetS919-SM and CUB-SM (curated with objects with small size), where the improvements from random cropping can be high up to almost 3 percent.

Reviewer comment (Experiments):
An essential prerequisite for the successful application of OWL-ViT in the GC-CLIP is that OWL-ViT can provide a detection box for every target in the image. The authors have yet to carry out validation on more datasets to verify the impact and constraints of the detectors on their method, so it cannot judge the processing performance of GC-CLIP for other more complex datasets, and more analysis is needed.

Author response:
**Response:** GC-CLIP can be viewed as a framework utilizing available object detectors to improve classification performance. In our work, we use OWL-ViT as a candidate of object detectors. We agree that the performance of OWL-ViT can be limited but we would like to show that it is good enough for improving classification performance under our framework. To demonstrate benefits of our setup in other scenarios, as suggested by the reviewer, we have conducted further experiments on additional common classification datasets (ImageNet, ImageNetV2 and Stanford Dogs) with their original test splits. The results are presented in the appendix A.9 in the updated version of our paper. According to the results, our Guided Cropping strategy can still consistently improve performance of the baselines.

Reviewer comment (Experiments):
Authors should report the performance comparison of GC-CLIP with current popular methods.

Author response:
**Response:** CALIP is actually one of the recent models from AAAI-2023 we consider in our work. Anyway, we also agree that including more baselines would make our results statistically more meaningful. In this regard, we incorporate a CLIP(ViT-L/14) variation trained with DataComp from NeurIPS-2023 [1] as one of our baselines. Its performance is reported in Table 3 of the appendix A.2 in the updated version of our paper. It must be noted that performance of DataComp can still be further improved by integrating with our Guided Cropping. This demonstrates consistent observations among various baselines.  

[1] "DATACOMP: In search of the next generation of multimodal datasets", NeurIPS, 2023.

Reviewer comment (Writing):
Authors also need to check for grammatical problems. For example, in the last sentence of paragraph 5 of the introduction section, there is a subject-verb inconsistency between “the cropped image” and “decrease” and “result in”.

Author response:
**Response:** Thank you for your comments. We have updated our text accordingly.

Reviewer comment (Experiments):
My main concern is the necessity of this guided cropping approach as it seems to have less difference in performance compared to what random cropping brings. There is a need for more results on more datasets and comparisons with other popular methods to demonstrate the performance of GC-CLIP.

Author response:
**Response:** We understand that your concern about slight improvement mainly refers to results on datasets with unconstrained object sizes (ImageNetS919 and CUB). In this case, this observation is expectable since many images of these datasets already have large enough object sizes (images have small extraneous regions) and thus do not greatly benefit from our Guided Cropping component. Note that our work is mainly motivated and designed to address the challenging cases of input images with small object sizes and large extraneous regions. Henceforth, our improvements are more evident on ImageNetS919-SM and CUB-SM (curated with objects with small size), where the improvements from random cropping can be high up to almost 3 percent.