Paper: BaFTA: Backprop-Free Test-Time Adaptation for Zero-shot Vision Language Models (ICLR-2024)

Reviewer comment (Novelty):
In the part of Projected Online Clustering, the main distinction between your approach and ReCLIP[1] is that your clustering is conducted online, while ReCLIP's clustering is done offline. Furthermore, due to your Test-Time adaptation setting, your clustering must be conducted online. So I think this may not represent a particularly solid innovation.

Author response:
ReCLIP focuses on source-free adaptation, employing a label propagation algorithm for pseudo label generation. However, this method relies on a substantial number of examples to yield satisfactory results, a condition that may not be met in many practical scenarios. In our exploration of alternatives to the label propagation approach, we found that online clustering performs satisfactorily. This approach alleviates the need for an extensive pool of available test data while maintaining performance standards to a reasonable degree.

Reviewer comment (Experiments):
The experiments are insufficient. In Table 3 and Table 4, I suggest that the authors include additional methods such as CoCoOp[2], TPT+CoCoOp, BaFTA+CoCoOp, and Sus-X [3] and [4].

Author response:
In response to your suggestion, we have incorporated additional methods (CoCoOp, CoCoOp+TPT, ProGrad, SuS-X) into the supplementary materials. However, we opted not to include CoCoOp+BaFTA for two reasons: 1) The authors of CoOp/CoCoOp have not released the weights for CoCoOp, preventing us from running BaFTA+CoCoOp; 2) CoOp+TPT outperforms CoCoOp+TPT on both RN50 and ViT-B/16, and we believe that comparing with the stronger CoOp+TPT is more informative. For a comprehensive examination of results and analysis, kindly refer to Section D and Table 4. We are planning to reorganize and integrate some of these methods into the main paper in future revisions.

Reviewer comment (Experiments):
The ablation study on entropy order α should be more comprehensive. In Table 7, when entropy order equals 0.25, the result is 70.52, and when it equals 0.5, the result is 70.53. So, what about when it equals 0.4 or 0.6? Therefore, the authors should conduct more fine-grained experiments.

Author response:
To offer more insights into the sensitivity of the Renyi Entropy parameter, we have performed extensive experiments with detailed results in the supplementary materials. The analysis suggests that BaFTA's performance is not highly sensitive to the choice of $\alpha$. Please refer to Section B, Figure 1 and Table 2 for a more detailed analysis.

Reviewer comment (Reproducibility):
The method proposed in this paper has persistence? In other words, can I save the results at a certain moment for later updates?

Author response:
Yes, BaFTA can be easily saved at any moment. The only evolving parameters are the moving averaged centroids and the cluster member counts, making them space-efficient for storage.

Reviewer comment (Theory):
In Table 6, the authors discuss the effectiveness of projection P*. Is there any explanation regarding how projection P* can improve the embedding distribution?

Author response:
The projection can be conceptually described in two steps: 1) SVD extracts the orthonormal basis U of the text embedding subspace. The projection matrix $P=UU^T$ projects everything onto the text embedding subspace, removing dimensions not participating in the classification calculation, since they are orthogonal to the text embeddings. This step removes the category agnostic information from visual embeddings and makes them easier to cluster; 2) $P^*=U’U’^T$ discards the first axis from U where all text embeddings overlap the most. This step removes shared information between text embeddings from different classes, emphasizing their differences and separating clusters from different classes.

Reviewer comment (Experiments):
A more comprehensive ablation of different components in the system is needed. In algorithm 1, the authors use $Eq.3$, online clustering, weighted aggregation, etc. I am confused about the contributions of different components. For example, in Table 5, the authors can add results of a simple average of BaFTA-OC + CLIP text predictions. This may help us understand the effectiveness of Renyi Aggregation more clearly. Other detailed ablations are also needed.

Author response:
In light of the concern regarding ablation experiments on various components of BaFTA, we've introduced additional ablation results in the supplementary materials. Specifically, we've included BaFTA-single, utilizing a single-template CLIP as the base model, and BaFTA-Avg, which employs a simple average function to merge predictions. Notably, the results from BaFTA-Avg exhibit a 1.68% decrease compared to the regular BaFTA, underscoring the effectiveness of Renyi Entropy Aggregation. For a comprehensive exploration of these results, please refer to Section C.1, Section C.2, and Table 3 in the supplementary materials.

Reviewer comment (Theory):
The core idea of the proposed method is to use visual centroids to help classifications. The online updating paradigm also assumes the test data come from the same distribution. In practice, this may not be the case. For instance, test samples may come from different distributions. One possible case is samples come from mixed distributions, e.g., mix of ImageNet-A&ImageNet-V2&ImageNet-R. I think the method will not work in such a circumstance.

Author response:
The consistent decent zero-shot classification accuracy achieved by CLIP across ImageNet and ImageNet-A/V2/W/R/Sketch using the same text embeddings generated from the CLIP text encoder suggests that the distributions of these datasets are not distinctly different from each other. Notably, ImageNet-R itself comprises a dataset with mixed distributions, including art, sketch, cartoon, plastic toys, etc. BaFTA demonstrates a significant improvement even in this mixed-distribution scenario. This indicates that BaFTA can effectively handle a certain level of distribution mixture, given that CLIP embeddings being sufficiently accurate, thereby ensuring that the distributions are not too dissimilar. It is possible to improve BaFTA with dynamic clustering mechanisms to handle an indefinite number of clusters or mixtures of severely different distributions. While this modification is beyond the scope of the current paper, we appreciate the suggestion and will consider it in future work.

Reviewer comment (Evaluation):
How about the inference time of the proposed method? The time cost of one step in Algorithm 1 from $for$ to $end for$.

Author response:
In response to the request for an inference time analysis, we have included a comparison of the inference time for BaFTA and TPT in our supplementary materials. The results indicate BaFTA is roughly 5 times faster than TPT. For detailed results and analysis, please refer to Section A and Table 1.

Reviewer comment (Evaluation):
Efficiency plays a pivotal role in test-time adaptation, as the ability to swiftly adapt to novel environments is of paramount importance. The paper should explicitly report and compare inference time metrics for BaFTA with existing methods like TPT. This can be done by measuring the time required for BaFTA to adapt to new data during inference and comparing it with the time taken by TPT or other relevant methods. Providing quantitative results and possibly a comparison table would be valuable for the readers.

Author response:
In response to the request for an efficiency analysis, we have included a detailed comparison of the inference time for BaFTA and TPT in our supplementary materials. Notably, BaFTA demonstrates an approximately 5 times speed improvement over TPT. For detailed results and analysis, please refer to Section A and Table 1.

Reviewer comment (Theory):
The paper assumes that the visual embeddings from CLIP are often discriminative enough for effective classification. A more detailed discussion or evidence regarding this assumption would strengthen the paper's argument.

Author response:
Regarding the assumption about CLIP visual embeddings, results in Table 2 of the main paper indicate that CLIP's linear evaluation results on all datasets are significantly higher than the corresponding zero-shot results. Linear evaluation is a protocol that assesses the quality of representations.  By freezing the CLIP visual embeddings and training a fully-supervised linear layer classifier, linear evaluation obtains the upper bound accuracy achievable by the frozen visual embeddings. Table 2 results suggest that the CLIP performance is likely constrained by the quality of text-generated classification weights. While CLIP visual embeddings are not perfect, we argue that, in the unsupervised test-time adaptation setting where only minor modifications can be made, focusing on text-side updates is more efficient due to the substantial improvement space available on the text side.

Reviewer comment (Experiments):
BaFTA's success seems to depend on parameters such as the projection space and the Renyi entropy threshold. The paper could provide more insights on how these parameters are chosen, tuned, and their sensitivity to results. A sensitivity analysis or parameter ablation study could be included.

Author response:
To offer more insights into the sensitivity of the Renyi Entropy parameter, we have added updated plots and results in the supplementary materials. The analysis suggests that BaFTA's performance is not highly sensitive to the choice of $\alpha$. Please refer to Section B, Figure 1, and Table 2 for a more detailed analysis. In terms of the embedding projection, it does not involve additional hyper-parameters except for the number of basis used to construct the projection matrix. For the majority of datasets, we set this number equal to the number of classes, considering it as the rank and the maximum number of basis for the text embedding subspace. However, for datasets with more than 150 categories, such as SUN397 and ImageNet, we observed that utilizing the top 150 basis is sufficient to capture most important dimensions from the embeddings. Therefore, we use the top 150 basis for SUN397 and ImageNet, as mentioned in the implementation detail section.

Reviewer comment (Experiments):
The experimental setting parameters are not the same with TPT and no fair experiment is conducted. The paper of Implementation Details shows: In line with the TPT implementation, we utilize a simple combination of RandomResizedCrop and RandomFlip to prepare 63 augmented views, constituting a mini-batch of 64 images for each test image. Actually, TPT only uses random resized crops to augmented views. Besides, “and adaptation In line with the TPT” seems less a “.”

Author response:
Regarding the implementation details, we carefully followed the official implementation from the [GitHub Repository](https://github.com/azshue/TPT/tree/main) provided in the TPT paper. In the actual implementation of TPT, both RandomResizedCrop and RandomFlip augmentations were used to produce the reported scores. Therefore, the comparison of BaFTA and TPT is fair.

Reviewer comment (Experiments):
Although the authors' approach can use multiple prompts compared to TPT only use single prompt, the paper lacks ablation experiments on prompt.

Author response:
In response to the concern about ablation experiments on prompts, we evaluated BaFTA with single-template CLIP as the base model. The BaFTA-single configuration resulted in a noteworthy 6.22% improvement in averaged accuracy across 15 datasets. You can find detailed results and analysis in Section C.1 and Table 3 of the supplementary materials.

Reviewer comment (Presentation):
Authors can present results in more forms of Effectiveness of Projected Embedding Space, such as t-SNE.

Author response:
To provide a more comprehensive understanding of the effectiveness of the projected embedding space, we have included t-SNE plots of CLIP visual embeddings in the supplementary materials. The t-SNE plot visually demonstrates that the projection space can effectively transform the sparse clusters into denser formation. Please refer to Section E and Figure 2 for detailed results and analysis.