Paper: AutoCLIP: Auto-tuning Zero-Shot Classifiers for Vision-Language Models (ICLR-2024)

Reviewer comment (Experiments):
The is one limitation: Current experiments suggest AutoCLIP can only be used for zero-shot classification, which is only one useful aspect of large vision and language model. Large vision and language model like CLIP is not about just doing zero-shot classification. For real applications, the impact of these models also lies in downstream tasks. For example, finetuning from CLIP pretrained parameters or using CLIP to directly assist the downstream tasks. It's better to explain some zero-shot classification applications.

Author response:
In this work, we do focus on zero-shot classification. We are aware that this is only one potential usage of VLMs. However, it is an important enough use-case that several prior works have focused on it, such as for instance DCLIP (Menon
& Vondrick, 2022) and WaffleCLIP (Roth et al., 2023). We also think that an improvement by 1 percent point accuracy is significant, particularly given that it comes without any retraining or custom prompt engineering by a pure change in the way the zero-shot classifier is constructed. However, we will further motivate zero-shot classification applications in the introduction as proposed by the reviewer.

Reviewer comment (Evaluation):
In view of the above limitations, another concern comes out about the result. If a model has 1% improvement on ImageNet, it may bring more improvements when the pre-trained model is applied in downstream task. In case AutoCLIP is not intended for downstream task, the 1% improvement may look not significant. What this 1% improvement can bring?

Author response:
In this work, we do focus on zero-shot classification. We are aware that this is only one potential usage of VLMs. However, it is an important enough use-case that several prior works have focused on it, such as for instance DCLIP (Menon
& Vondrick, 2022) and WaffleCLIP (Roth et al., 2023). We also think that an improvement by 1 percent point accuracy is significant, particularly given that it comes without any retraining or custom prompt engineering by a pure change in the way the zero-shot classifier is constructed. However, we will further motivate zero-shot classification applications in the introduction as proposed by the reviewer.

Reviewer comment (Evaluation):
The experiment in fig.3 is not clear. What does this experiment verify for? Any explain why ViT-L is worse? Does it mean AutoCLIP cannot well be scaled to larger models?

Author response:
Figure 2 provides ample evidence that AutoCLIP also brings benefits for larger models (ViT-L-14s) on multiple datasets. Figure 3 shows that in the specific combination of large models and image corruptions, AutoCLIP does not bring benefits compared to baseline zero-shot classifiers. But still, for 3 out of the 5 VLMs in Figure 3 (and averaged across models), AutoCLIP is better than the baseline zero-shot classifier.

Reviewer comment (Experiments):
It is highly suggested to show evidence for "This is inspired by the observation that class descriptors having higher similarity in the embedding space describe the image better". This claim support the rationale of the proposed AutoCLIP. To my understanding, if this claim does not hold, then AutoCLIP does not hold.

Author response:
We provide evidence for this in the paper in Figure 6 and in Figure 7 (in the appendix): Figure 6 shows that on images from the Food dataset, prompts starting with “A photo of...” get higher weights by AutoCLIP than prompts starting with "Art of ...", while on ImageNetR in Figure 7, it is the other way around. This corresponds to the Food dataset containing actual photos while ImageNetR containing more artistic renditions etc. Thus higher weights correspond to better class descriptor-image fit.

Reviewer comment (Evaluation):
Is there any speed comparison? How much more inference time does the AutoCLIP need?

Author response:
We provide some numbers for the case of a ViT-L-14 on the Oxford Pets dataset. Here, encoding an image takes 12.64ms on a V100 (minimum over 100 images). The baseline "averaging" zero-shot classifiers takes additional 0.08ms (average over 640 samples) on top to classify a sample. AutoCLIP takes additional 1.54ms (average over 640 samples) for classification when running bisection for autotuning the step size. For a fixed step size, the overhead of AutoCLIP is only 0.45ms. Thus, AutoCLIP raises inference from 12.64ms to a bit more than 14ms. In contrast, TPT (Shu et al., 2022) and  RLCF (Zhao et al., 2023), which did not report compute or memory requirements, require encoding multiple  image augmentations. TPT states "We augment a single test image 63 times using random resized crops and construct a batch of 64 images, including the original one.", which means that the image encoding time (for instance the 12.64ms from above) is increased by a factor of 64x, plus additional overhead for backpropagating through the text encoder, which likely brings the inference time per sample close to 1s. RLCF also requires multiple image augmentation but does not state the number of augmentations used. In contrast, AutoCLIP does not require any image augmentations or backpropagating through the text encoder and increase inference time from 12.64ms to ~14.0ms instead of ~1 second.

Reviewer comment (Novelty):
The main concern of this work is its novelty. The idea of weighing the text features from different prompts has been proposed [a]. It is unclear why is the proposed method different than [a].

Author response:
We thank the reviewer for bringing the work [a] to our attention; we will include a discussion of it in the related works in the next revision of our paper. We note that ZPE (the method from [a]) has more restrictive prerequisites than our AutoCLIP: The "max logit scoring" of ZPE determines prompt weights based on logit statistics in the target domain, which must be computed from multiple images from the target domain. This limits ZPE to operate only with a batch of images instead of a single image. Additionally, the "logit normalization" of ZPE requires availability of image features which represent the feature distribution in the pre-training domain, that is: ZPE with logit normalization is not "source-free". In contrast, our AutoCLIP requires no additional statistics so that we can do the inference with AutoCLIP in a source-free setting from a single input image alone. We note that ZPE and AutoCLIP are orthogonal and could be combined: first normalize logits with ZPE and then construct a auto-tuned zero-shot classifier with AutoCLIP on top.

Reviewer comment (Experiments):
While adopting the weighting strategy in [a] during test time is a solution for boosting the classification performance, one could simply average the topK text prompts that have the topK cosine similarity with the image feature. Is the proposed method better?

Author response:
Thanks for proposing TopR as a baseline (we denote the number of selected queries by R to avoid confusion with the number of total prompt templates that we denote by K in the paper). It is indeed a stronger baseline than max or mean aggregation for suitably chosen R (we note that choosing R is non-trivial in a zero-shot setting). We have run an additional study comparing AutoCLIP against this TopR-CLIP for $100$ DCLIP prompt template, across the same VLMs and datasets as in Figure 2. We provide results for different choices of R, both in terms of the percentage of settings in which AutoCLIP is better as well as the Δ Accuracy compared to TopR-CLIP below.

| TopR | AutoCLIP better | Δ Accuracy
| ----------- | ----------- | ----------- |
|  1 | 100% | 5.56 |
|  5 | 100% | 1.61 |
| 10 | 92% | 0.79 |
| 20 | 86% | 0.40 |
| 50 | 89% | 0.27 |
|100 | 94% | 0.51 |

We note that even for the optimal choice of R (R=20), which would require tuning R on labeled data, AutoCLIP is better than TopR-CLIP in 86\% of the cases. We will include a more detailed visualization of this study in the appendix of the revised version of the paper.

Reviewer comment (Presentation):
The author is suggested to put the argument explanation of scipy.optimize.bisect to the appendix.

Author response:
Thanks for the suggestion, we will do so.

Reviewer comment (Presentation):
The x-axis of Figure 2 is not clear. Why are there so many lines on the x-axis? Are they meaningful?

Author response:
These are just the default ticks of matplotlib for a logarithmic x-axis. We did evaluate for $K \in$ {2, 4, 10, 20, 50, 100, 200, 400, 1000} prompt templates in this figure.

Reviewer comment (Writing):
Unclear: “from ∆ = 0.06 for K = 4 over ∆ = 0.33 for K = 10 and ∆ = 0.49 for K = 50 to ∆ = 0.57 for K = 200”.

Author response:
Here K denotes the number of prompt templates sampled from CLIP/DCLIP/WaffleCLIP and ∆ the average increase in accuracy when using AutoCLIP compared to the baseline "averaging" zero-shot classifier. Did this explanation clarify the statement?

Reviewer comment (Novelty):
The author is suggested to explain the fundamental differences between this work and [a].

Author response:
We thank the reviewer for bringing the work [a] to our attention; we will include a discussion of it in the related works in the next revision of our paper. We note that ZPE (the method from [a]) has more restrictive prerequisites than our AutoCLIP: The "max logit scoring" of ZPE determines prompt weights based on logit statistics in the target domain, which must be computed from multiple images from the target domain. This limits ZPE to operate only with a batch of images instead of a single image. Additionally, the "logit normalization" of ZPE requires availability of image features which represent the feature distribution in the pre-training domain, that is: ZPE with logit normalization is not "source-free". In contrast, our AutoCLIP requires no additional statistics so that we can do the inference with AutoCLIP in a source-free setting from a single input image alone. We note that ZPE and AutoCLIP are orthogonal and could be combined: first normalize logits with ZPE and then construct a auto-tuned zero-shot classifier with AutoCLIP on top.

Reviewer comment (Experiments):
The author is suggested to show that the proposed method is better than averaging the topK text prompts that have the topK cosine similarity. When K=1, this is equivalent to ``max” aggregation and when K=80 (e.g., for CLIP), which means all the text prompts are averaged. Please ablate the performance for different K.

Author response:
Thanks for proposing TopR as a baseline (we denote the number of selected queries by R to avoid confusion with the number of total prompt templates that we denote by K in the paper). It is indeed a stronger baseline than max or mean aggregation for suitably chosen R (we note that choosing R is non-trivial in a zero-shot setting). We have run an additional study comparing AutoCLIP against this TopR-CLIP for $100$ DCLIP prompt template, across the same VLMs and datasets as in Figure 2. We provide results for different choices of R, both in terms of the percentage of settings in which AutoCLIP is better as well as the Δ Accuracy compared to TopR-CLIP below.

| TopR | AutoCLIP better | Δ Accuracy
| ----------- | ----------- | ----------- |
|  1 | 100% | 5.56 |
|  5 | 100% | 1.61 |
| 10 | 92% | 0.79 |
| 20 | 86% | 0.40 |
| 50 | 89% | 0.27 |
|100 | 94% | 0.51 |

We note that even for the optimal choice of R (R=20), which would require tuning R on labeled data, AutoCLIP is better than TopR-CLIP in 86\% of the cases. We will include a more detailed visualization of this study in the appendix of the revised version of the paper.

Reviewer comment (Novelty):
Even though the proposed method is simple and effective, the method looks like a naive modification method to replace the uniform-weighted average descriptor encodings with weighted average encodings based on the existing algorithms which may limit the paper's novelty.

Author response:
We agree that AutoCLIP is simple and effective; however, we do not think AutoCLIP is naive since it is a principled solution (gradient-based maximization of an objective) based on a clear intuition: prompt templates resulting in class descriptors more similar to the image on average should get higher weights. Being a simple method is in our opinion a feature not a bug of our contribution.

Reviewer comment (Reproducibility):
I'd like to know if the order of the queries is different, will the performance remain the same? If each query has different classes in the inference time, will the weighted average encodings need to be changed/calculated all the time?

Author response:
AutoCLIP is by design permutation-invariant, that is: it treats queries as a set and order of queries does not matter. We also run an empirical sanity check and randomly permuting queries did not change performance.

Reviewer comment (Evaluation):
Similarly, will the proposed method still have the same speed in the inference process or it will be dramatically different?

Author response:
AutoCLIP zero-shot classifiers have overhead at inference time compared to standard zero-shot classifiers that simply average encoded class descriptors. However, this overhead is still very small compared to the encoding of the image with the VLM's image encoder in most cases. For instance, in the case of a ViT-L-14 on the Oxford Pets dataset, encoding an image takes 12.64ms on a V100 (minimum over 100 images). The baseline "averaging" zero-shot classifiers takes additional 0.08ms (average over 640 samples) on top to classify a sample. AutoCLIP takes additional 1.54ms (average over 640 samples) for classification when running bisection for autotuning the step size. For a fixed step size, the overhead of AutoCLIP is only 0.45ms. We note that AutoCLIP's overhead is very small compared to other VLM test-time adaption approaches (see below).

Reviewer comment (Novelty):
Minor: with the logsumexp providing the best result compared to entropy, mean, and max, is the idea similar to focal loss with multi-class?

Author response:
Relating AutoCLIP's logsumexp objective function to other losses is interesting. We do not see an immediate connection to focal loss (maybe the reviewer can provide further details on the possible relationship?). However, as stated in the paper we note that −logsumexp has been interpreted as the energy function of a data point (for appropriately trained classifiers) (Grathwohl et al., 2020); in this view, AutoCLIP can be interpreted as minimizing the energy and maximizing the probability density p(x) of x under the zero-shot classifier.