Paper: PRO: Pseudo-label Regularized Optimization on Unlabeled Test Data (ICLR-2024)

Reviewer comment (Theory):
While this paper empirically validates which setups/parameters are useful for test time training, it does not provide theoretical understanding on why they do not collapse.

Author response:
We note that imposing a penalty against deviance from the initial label distribution intuitively discourages collapsed solutions. The primary aim of this paper is to conduct an empirical investigation to substantiate this intuition and explore methods useful for test-time training. The potential for theoretical exploration into this aspect is left for future works.

Reviewer comment (Reproducibility):
Details of algorithm (PRO) are missing. It is unclear how the loss functions are combined.

Author response:
We will add equations detailing each loss function in the updated draft.

Reviewer comment (Reproducibility):
Can all experiment/analysis results of section 5 be included in appendix? While Section 5 contains various insights from experiments, full experiment setup and results are omitted.

Author response:
We will expand on the discussion on the full experiments in the appendix in the updated version.

Reviewer comment (Evaluation):
What’s the failure mode of the suggested method other than class imbalance?

Author response:
As shown in Figure. 3, we do not observe any significant failure mode of the suggested method.

Reviewer comment (Experiments):
Can it be shown that PRO is effective in setups other than CLIP?

Author response:
Our primary focus lies in understanding the distinctive distribution shifts associated with the adaptation of CLIP models in zero-shot tasks. While PRO may demonstrate effectiveness in handling other shifts and architectures, we consider these settings more appropriate for exploration in future works.

Reviewer comment (Experiments):
The authors have neglected to benchmark PRO against other related test-time adaptation methods employing semi-supervised learning strategies, such as self-training [2].

Author response:
We thank the reviewer for bringing relevant works to our attention. However, we note that many test-time adaptation methods, including the one reviewer suggested, are designed for non-CLIP models. At a fundamental level, the distribution shifts addressed in these works differ substantially from those encountered in our context. ResNet models, for example, are trained on supervised classification tasks with training data resembling the test data. CLIP models, however, were trained via contrastive losses on data that may be significantly different or irrelevant to the downstream data we aim to adapt to. Additionally, it is unclear whether the adaptation should be applied to either or both of the visual and textual towers of CLIP, and improbable that the hyperparameter choices are transferable. 

We remark that comparison of PRO against a test-time adaptation method developed for CLIP (MUST) is provided in Table 1, where we find significant advantage in PRO over MUST. We additionally implement test-time prompt tuning (TPT) of Shu et. al [1] and found that on ViT-B/32, TPT results in 43% and 64% in accuracies for DTD and UCF-101 datasets, which are correspondingly 6% and 3% lower than the performance of PRO-CLIP with textual layer (see Table. 1)

Reviewer comment (Evaluation):
The efficacy of PRO seems limited to specific architectures; it yields noticeable improvements when applied to VIT models but is virtually ineffective with ResNet architectures, resulting in a marginal improvement of only 0.1% (See Appendix B).

Author response:
We appreciate the reviewer's keen observation. It's important to highlight that the distribution shifts examined in Appendix B with the ResNet architecture differ significantly from those addressed in the main text with CLIP. While we acknowledge that PRO might be more suited for the latter scenario, we emphasize that as foundation models like CLIP gain widespread use, the latter setting becomes increasingly relevant.

Reviewer comment (Experiments):
The experimental results of the paper are not convincing enough. First of all, PRO-CLIP in Figure 3 has little performance gains on commonly used image datasets. Secondly, PRO is a cherry-picked combination of existing methods. Without the support of convincing experimental results, it is difficult to verify its effectiveness. The authors could consider performing extensive experiments in additional text modalities, which I believe is fully achievable with a CLIP-based approach.

Author response:
>**The experimental results of the paper are not convincing enough. First of all, PRO-CLIP in Figure 3 has little performance gains on commonly used image datasets**  

We'd like to highlight that PRO-CLIP leads to an average accuracy improvement of 2.5%, with a maximum enhancement of 6.1% for CLIP. Unlike prior works, we do not perform hyperparameter selection by peeking at the resulting accuracies of individual datasets. Instead, we adopt a consistent set of hyperparameters and conduct careful experiments to avoid meta-overfitting, thereby ensuring the robustness of our approach. We substantiated the robustness of our method in Sec. 6.2, where we find that PRO-CLIP with the same hyperparameter selection results in significant gain (up to 7.8% increase in accuracy) on additional, previously unseen datasets.

Reviewer comment (Evaluation):
My biggest concern is the lack of mean and standard deviation in the reported results to see if the results are significant. This would require repeating the experiments multiple times.

Author response:
We wish to emphasize that our approach lacks inherent randomness, as a result we are not able to calculate standard deviation. We remark that the experimental results show that our proposed method consistently enhances the accuracy of CLIP over 18 datasets, and hope that this addresses the reviewer’s concern on the robustness of our method.

Reviewer comment (Experiments):
Another concern is the lack of generalization to other tasks like image segmentation and text-related problems which have become high in demand these days.

Author response:
We agree with the reviewer that these are essential tasks for test-time adaptation. As an initial evaluation of the proposed method, we have chosen to concentrate on image classification tasks as a starting point. The investigation of appropriate methods for test-time adaptation in other tasks is deferred to future work.