Paper: Towards Realistic Unsupervised Fine-tuning with Vision-Language Models (ICLR-2024)

Reviewer comment (Theory):
One potential weakness of the paper is that it relies on sample-level confidence weights to approximate entropy minimization and maximization. While this approach is innovative, it may be sensitive to the distribution of confidences within the unlabeled data. If the confidences are not well-calibrated or vary significantly across samples, it could affect the effectiveness of UEO. The performance of UEO might be influenced by the quality and reliability of the confidence estimates, and if the confidence estimates are noisy or inaccurate, it could lead to suboptimal results.

Author response:
**[A1]**. We agree on the significance of the instance weight $w(x)$. However, UEO has been empirically demonstrated to be not sensitive to confidences. To assess the calibration of instance weights, we utilize the AUC score as an indicator. For instance, when comparing the results on the DN (Avg.) dataset under open-partial shift with two different backbones (ResNet-50 in Table 4 and ViT-B/16 in Table 5), UEO consistently outperforms the CLIP baseline in terms of the accuracy (ACC) score. It is worth noting that improved calibration may not necessarily result in larger gains. Similar observations can be made when comparing different datasets under the same backbone.

|  DN (Avg.) [open-partial]    | ACC  | AUC  |
|----------------|------|------|
| CLIP (ResNet-50) | 47.3 | 66.5 |
| UEO (ResNet-50)  | 51.9 | 67.4 |
| CLIP (ViT-B/16) | 58.2 | 72.6 |
| UEO (ViT-B/16)  | 62.0 | 72.9 |

Reviewer comment (Experiments):
One experiment that is missing would involve applying the method to VLMs that are not well-calibrated and observing its impact on the performance of the fine-tuned model.

Author response:
**[A2]**. As obtaining a poorly calibrated VLM is challenging, we kindly refer the reviewer to **[A5] @ Reviewer eDVD** for a visual pre-trained source model (VM). Carefully comparing the gains of UEO over the initized model in two following tables, we find that the AUC scores vary a lot across different models, and UEO always increases in terms of the ACC score.

|  OF(A) [open-partial]  | ACC  | AUC  |
|----------------|------|------|
| VM | 65.2 | 71.8 |
| UEO(VM)| 70.2 |	74.1 |
| CLIP| 76.6 | 84.8 |
| UEO(CLIP)  | 80.2 | 84.5 |

|  OH(A) [open-partial]  | ACC  | AUC  |
|----------------|------|------|
| VM | 54.5	|64.5 |
| UEO(VM)| 58.2 |	64.7 |
| CLIP| 70.2 |74.9 |
| UEO(CLIP)  | 72.3 | 75.0 |

Reviewer comment (Evaluation):
Another potential drawback is that, when compared to InfoMax, there does not appear to be a significant improvement in accuracy on certain datasets. For instance, on the DomainNet dataset, there is no noticeable difference in performance in accuracy.

Author response:
**[A3]**. We respectfully disagree. As stated in **[A2] @ Reviewer pTa8**, UEO consistently outperforms InfoMax when considering both accuracy (ACC) and area under the curve (AUC) together. While the difference between UEO and InfoMax on the DomainNet dataset may appear small, UEO consistently exhibits superior AUC scores under different category shifts and backbones.

Reviewer comment (Experiments):
As highlighted in the introduction and method sections, the paper's primary focus is the class discrepancy between unlabeled data and the predefined label space. However, the principal experiments are based on domain adaptation datasets, characterized predominantly by distributional differences between domains. I believe more general classification datasets (e.g., ImageNet and SUN397 as in CoOp) should be employed to define the task setting and verify the efficacy of the proposed method.

Author response:
**[A1]**. A great advice. To validate the effectiveness of the proposed method (UEO), we additionally utilize three widely recognized classification datasets (i.e., ImageNet, SUN397, and Food101) and present the results under four category shifts in the tables below.

| ImageNet | ACC (C) | AUC (C) | ACC (P) | AUC (P) | ACC (O) | AUC (O) | ACC (OP) | AUC (OP) | ACC (Avg.) | AUC (Avg.) |
|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|
| CLIP    | 63.8 | 73.8 | 63.8 | 73.8 | 63.8 | 73.8 | 63.8 | 73.8    | 63.8 | 73.8 |
| UPL     | 65.6 | 73.6 | 65.2 | 73.9 | 65.8 | 73.5 | 65.6 | 72.9    | 65.5 | 73.5 |
| POUF    | 65.7 | 72.0 | 65.2 | 72.8 | 65.7 | 71.0 | 65.8 | 71.7    | 65.6 | 71.9 |
| DANCE   | 64.7 | 72.8 | 64.3 | 73.3 | 64.9 | 72.1 | 64.4 | 72.6    | 64.6 | 72.7 |
| EntMin  | 64.0 | 73.5 | 64.1 | 73.8 | 63.9 | 73.1 | 63.8 | 73.3    | 63.9 | 73.4 |
| InfoMax | 66.3 | 71.8 | 65.4 | 73.4 | 65.1 | 64.5 | 65.3 | 68.3    | 65.5 | 69.5 |
| UEO     | 65.9 | 73.0 | 65.3 | 74.2 | 65.9 | 72.4 | 65.9 | 73.2    | 65.7 | 73.2 |

| SUN397 | ACC (C) | AUC (C) | ACC (P) | AUC (P) | ACC (O) | AUC (O) | ACC (OP) | AUC (OP) | ACC (Avg.) | AUC (Avg.) |
|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|
| CLIP    | 63.7 | 68.6 | 63.7 | 68.6 | 63.7 | 68.6 | 63.7 | 68.6    | 63.7 | 68.6 |
| UPL     | 66.0 | 69.0 | 65.6 | 69.0 | 66.2 | 69.1 | 66.0 | 69.0    | 65.9 | 69.0 |
| POUF    | 66.2 | 69.0 | 65.9 | 69.0 | 66.2 | 68.9 | 66.1 | 69.0    | 66.1 | 69.0 |
| DANCE   | 64.8 | 68.9 | 64.6 | 68.8 | 64.9 | 68.9 | 64.8 | 68.8    | 64.8 | 68.8 |
| EntMin  | 64.1 | 68.7 | 64.0 | 68.6 | 64.1 | 68.7 | 64.1 | 68.6    | 64.1 | 68.6 |
| InfoMax | 67.0 | 68.7 | 66.8 | 68.8 | 67.0 | 68.5 | 67.0 | 68.6    | 67.0 | 68.7 |
| UEO     | 66.5 | 68.9 | 66.4 | 69.0 | 66.7 | 68.8 | 66.6 | 68.8    | 66.5 | 68.9 |

| Food101 | ACC (C) | AUC (C) | ACC (P) | AUC (P) | ACC (O) | AUC (O) | ACC (OP) | AUC (OP) | ACC (Avg.) | AUC (Avg.) |
|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|
| CLIP    | 79.7 | 77.6 | 79.7 | 77.6 | 79.7 | 77.6 | 79.7 | 77.6    | 79.7 | 77.6 |
| UPL     | 80.2 | 77.7 | 80.6 | 77.3 | 80.3 | 78.0 | 81.2 | 78.4    | 80.6 | 77.8 |
| POUF    | 81.5 | 77.6 | 81.1 | 77.6 | 81.5 | 77.2 | 81.3 | 77.2    | 81.3 | 77.4 |
| DANCE   | 81.0 | 77.4 | 80.4 | 77.4 | 80.4 | 76.3 | 80.0 | 76.2    | 80.4 | 76.8 |
| EntMin  | 80.1 | 77.6 | 79.9 | 77.6 | 80.0 | 77.2 | 79.9 | 77.1    | 80.0 | 77.4 |
| InfoMax | 82.5 | 77.1 | 82.1 | 77.7 | 82.6 | 76.5 | 82.3 | 76.0    | 82.4 | 76.8 |
| UEO     | 82.4 | 77.9 | 81.9 | 77.9 | 82.3 | 77.4 | 82.1 | 77.7    | 82.2 | 77.7 |

[(C): closed-set, (P): partial-set, (O): open-set, (OP): open-partial]

On the ImageNet dataset, UEO and UPL are top two methods when considering both ACC and AUC simultaneously. Similarly, UEO and InfoMax are top two methods on the SUN397 dataset. Notably, UEO achieves the best performance on the FOOD101 dataset. In summary, UEO consistently attains the best or competitive performance across different datasets, while other methods exhibit more variability across the datasets.

Reviewer comment (Experiments):
Missing ablation studies. The methodology encompasses two critical components: firstly, the entropy-related training objective, and secondly, the parameter-efficient tuning. I'm uncertain if the competing methods in Tab. 1 also implement the same parameter-efficient tuning. Furthermore, the prompt and affine parameters of the parameter-efficient tuning should be dissected and examined individually.

Author response:
**[A3]**. We feel sorry for any confusion. As written in the last paragrpah in Sec 4.1, all the methods are implmented with the same parameter-efficient tuning techique. In particular, we reduce the learning rate for EntMin due to its rapidly decreasing results.

To verify the effectiveness of the proposed parameter-efficient tuning strategy, we present results with only text prompt tuning in the Appendix (Tables 14-17). Additionally, we investigate the influence of **different combinations of parameters during the fine-tuning process in Fig. 3**. In these figures, the first two columns denote optimizing the text prompt and visual affine parameters, respectively.

Reviewer comment (Novelty):
Missing references. Some related studies delve into various task settings, such as black-box DA in [1], partial DA in [2], and universal DA in [3].  

[1] Unsupervised Domain Adaptation of Black-Box Source Models, BMVC21  
[2] Universal domain adaptation, CVPR19  
[3] Partial adversarial domain adaptation, ECCV18

Author response:
**[A4]**. Thanks for your kind reminder. [r1] explores the model adaptation problem in a black-box manner, [r2] considers domain adaptation under open-partial category shift, and [r3] addresses domain adaptation under partial-set category shift. These papers represent various domain adaptation problem setups, and we will include them in the related works section. Given that these methods are tailored to specific category shifts, which significantly differ from the problem setup presented here, direct comparisons with UEO in the experiments prove challenging.

- [r1] Zhang, Haojian, et al. "Unsupervised Domain Adaptation of Black-Box Source Models." In Proc. BMVC 2021.
- [r2] You, Kaichao, et al. "Universal domain adaptation." Proc. CVPR. 2019.
- [r3] Cao, Zhangjie, et al. "Partial adversarial domain adaptation." Proc. ECCV. 2018.

Reviewer comment (Experiments):
The authors should better clarify how their experimental setting differs from previous works. Specifically, they mentioned "unsupervised universal fine-tuning" as a novel fine-tuning setup, but the evaluation appears to directly adopt previous datasets without special curation. It is unclear whether this is a new setting, an adaptation of a previous setting to CLIP, or some other case.

Author response:
**[A2]**. We feel sorry for any confusion. We would like to clarify that the proposed unsupervised universal fine-tuning problem **is not derived from other related topics but constitutes a new and intriguing task**. Since the proposed task is closely related to several tasks in the domain adaptation area (i.e., model adaptation and universal domain adaptation), we employ several popular benchmarks (e.g., Office, OfficeHome, and DomainNet) in the domain adaptation area. We provide additional results on more general benchmarks in **[A1] @ Reviewer pTa8**.

To the best of our knowledge, our task represents the first exploration into both ID generalization and OOD detection for unsupervised fine-tuning under unknown category shifts. Unlike most prior domain adaptation methods that assume knowledge of the category shift in advance and predominantly emphasize the ability of ID generalization, our task is more realistic. In the context of universal domain adaptation tasks, where the target data for adaptation and the test data share the same label space, evaluating the ability of OOD detection becomes challenging (as depcited in Evaluation of Sec. 3.2). Instead, our task introduces a distinct fine-tuning and evaluation protocol, along with different evaluation metrics.

Reviewer comment (Experiments):
The authors should clarify whether there is a clear baseline for the UEO approach, for example, some simple modification or fine-tuning strategy to CLIP for this setting.

Author response:
**[A3]**. For each question, we offer a point-to-point response below.

- For the proposed parameter-tuning stragey, we show the results of only prompt tuning in Fig. 3 and Tables 14 - 17 in the Appendix. Regarding the objectives, EntMin and InfoMax could be considered as two baselines for UEO. Further comparisons with two-step thresholding variants are elaborated in **[A4] @ Reviewer eDVD**.
- Regarding the variance of the numbers, we conducted all the experiments using different seeds and observed a relatively small variance, approximately 0.1 for both ACC and AUC. Therefore, we omitted the variance in these tables
- Generally, DomainNet could be considered as the largest and hardest dataset in this paper. For instance, the Real domain in the DomainNet dataset comprises approximately 120k images across 345 classes. Notably, the ACC and AUC scores of the original CLIP model are consistently the lowest among all the datasets.
- In some datasets, UEO might not exhibit a clear superiority over the baselines, primarily for two reasons. Firstly, unsupervised fine-tuning is inherently challenging, and the majority of fine-tuning methods do not significantly outperform the original CLIP model. Secondly, enhancing both ACC and AUC scores concurrently is a difficult task. In most situations, a trade-off exists between these two scores. Notably, UEO consistently improves the ACC score without compromising the AUC score, outperforming other methods across the majority of datasets and category shifts.

Reviewer comment (Evaluation):
The variance of the reported numbers should be provided.

Author response:
**[A3]**. For each question, we offer a point-to-point response below.

- For the proposed parameter-tuning stragey, we show the results of only prompt tuning in Fig. 3 and Tables 14 - 17 in the Appendix. Regarding the objectives, EntMin and InfoMax could be considered as two baselines for UEO. Further comparisons with two-step thresholding variants are elaborated in **[A4] @ Reviewer eDVD**.
- Regarding the variance of the numbers, we conducted all the experiments using different seeds and observed a relatively small variance, approximately 0.1 for both ACC and AUC. Therefore, we omitted the variance in these tables
- Generally, DomainNet could be considered as the largest and hardest dataset in this paper. For instance, the Real domain in the DomainNet dataset comprises approximately 120k images across 345 classes. Notably, the ACC and AUC scores of the original CLIP model are consistently the lowest among all the datasets.
- In some datasets, UEO might not exhibit a clear superiority over the baselines, primarily for two reasons. Firstly, unsupervised fine-tuning is inherently challenging, and the majority of fine-tuning methods do not significantly outperform the original CLIP model. Secondly, enhancing both ACC and AUC scores concurrently is a difficult task. In most situations, a trade-off exists between these two scores. Notably, UEO consistently improves the ACC score without compromising the AUC score, outperforming other methods across the majority of datasets and category shifts.

Reviewer comment (Experiments):
The authors should identify which is the largest and hardest dataset.

Author response:
**[A3]**. For each question, we offer a point-to-point response below.

- For the proposed parameter-tuning stragey, we show the results of only prompt tuning in Fig. 3 and Tables 14 - 17 in the Appendix. Regarding the objectives, EntMin and InfoMax could be considered as two baselines for UEO. Further comparisons with two-step thresholding variants are elaborated in **[A4] @ Reviewer eDVD**.
- Regarding the variance of the numbers, we conducted all the experiments using different seeds and observed a relatively small variance, approximately 0.1 for both ACC and AUC. Therefore, we omitted the variance in these tables
- Generally, DomainNet could be considered as the largest and hardest dataset in this paper. For instance, the Real domain in the DomainNet dataset comprises approximately 120k images across 345 classes. Notably, the ACC and AUC scores of the original CLIP model are consistently the lowest among all the datasets.
- In some datasets, UEO might not exhibit a clear superiority over the baselines, primarily for two reasons. Firstly, unsupervised fine-tuning is inherently challenging, and the majority of fine-tuning methods do not significantly outperform the original CLIP model. Secondly, enhancing both ACC and AUC scores concurrently is a difficult task. In most situations, a trade-off exists between these two scores. Notably, UEO consistently improves the ACC score without compromising the AUC score, outperforming other methods across the majority of datasets and category shifts.

Reviewer comment (Evaluation):
Although state-of-the-art performance is not necessary, the authors should investigate the difference in performance and offer insights. For example, in Table 1, the gap between UEO and UPL on OH (Avg.) is quite significant, and the cause of this difference should be discussed.

Author response:
**[A3]**. For each question, we offer a point-to-point response below.

- For the proposed parameter-tuning stragey, we show the results of only prompt tuning in Fig. 3 and Tables 14 - 17 in the Appendix. Regarding the objectives, EntMin and InfoMax could be considered as two baselines for UEO. Further comparisons with two-step thresholding variants are elaborated in **[A4] @ Reviewer eDVD**.
- Regarding the variance of the numbers, we conducted all the experiments using different seeds and observed a relatively small variance, approximately 0.1 for both ACC and AUC. Therefore, we omitted the variance in these tables
- Generally, DomainNet could be considered as the largest and hardest dataset in this paper. For instance, the Real domain in the DomainNet dataset comprises approximately 120k images across 345 classes. Notably, the ACC and AUC scores of the original CLIP model are consistently the lowest among all the datasets.
- In some datasets, UEO might not exhibit a clear superiority over the baselines, primarily for two reasons. Firstly, unsupervised fine-tuning is inherently challenging, and the majority of fine-tuning methods do not significantly outperform the original CLIP model. Secondly, enhancing both ACC and AUC scores concurrently is a difficult task. In most situations, a trade-off exists between these two scores. Notably, UEO consistently improves the ACC score without compromising the AUC score, outperforming other methods across the majority of datasets and category shifts.

Reviewer comment (Novelty):
The strategy of entropy minimization for ID samples and entropy maximization for OOD samples have been a popular method[1, 2, 3]. This study applies the principle to the field of vision-language models. Despite its effectiveness on different benchmarks, the core idea resembles traditional ones, which would compromise the novelty of this study.

Author response:
We thank the reviewer for providing valuable comments and address your concerns in the following responses.

> **[Q1]**. The resemblance to previous methods [1,2,3] would compromise the novelty of this study.

**[A1]**. After carefully reviewing these papers [1,2,3], we observe that each of them utilizes a similar strategy, aiming to minimize the entropy of ID samples and maximize the entropy of OOD samples to enhance OOD detection capability. While our method does share a similar high-level idea, it diverges from these approaches in two key aspects.

**The first difference lies in the problem setup**, as our fine-tuning process is executed solely with unlabeled data, without explicit ID and OOD samples. In certain category shifts, such as closed-set and partial-set scenarios, OOD samples may not even exist in the unlabeled data, a condition that was not feasible for previous methods.

**The second difference lies in the ultimate objective**. If we know $w(x)\to 1$ for ID samples and $w(x)\to 0$ for OOD samples for **unlabeled** data in advance, Eq. (3) exhibits similarities to prior OOD detection methods [1,2,3]. However, our proposed method optimizes the objective in Eq. (4), representing a more relaxed yet robust alternative to Eq. (3).

Reviewer comment (Novelty):
I understand that the authors contribute in a generalized form as in Eq. (3) & (4) for the loss function of both ID and OOD samples. However, a similar principle of maximizing Mutual Information ID samples and penalizing the mutual information of OOD samples has been also proposed in [4].

Author response:
> **[Q2]**. However, a similar principle of maximizing Mutual Information ID samples and penalizing the mutual information of OOD samples has been also proposed in [4].

**[A2]**. Apologies for any confusion. Our method is different from [4]. Firstly, the method in [4] adopts the meta-learning framework that splits ID and OOD samples in the meta-training step, retaining the ground truth knowledge of ID and OOD during training. Secondly, the Mutual Information Maximization training objective in [4] is defined as the binary cross-entropy loss between ID and OOD prediction, which is distinct from our entropic loss. To be honest, minimizing the scores for ID samples and maximizing the scores for OOD samples a straightforward solution given the presence of ID and OOD samples in the training process. However, in the context of unsupervised fine-tuning, a key challenge arises in distinguishing ID from OOD samples. To circumvent the need for sensitive thresholding operations, we propose the relaxed optimization objective in Eq. (4).

Beyond the unified objective in Eq. (4), our paper introduces **a novel unsupervised fine-tuning problem setup** that considers both ID generalization and OOD detection. This is especially crucial in handling various category shifts between the training data and the pre-defined label set, making our fine-tuning approach a notable contribution to the field.
particularly in the face of various category shifts between the training data and the pre-defined label set. We consider such a fine-tuning problem setup to be a significant contribution to the field. Additionally, we provide **a new parameter-efficient fine-tuning strategy** for CLIP that optimizes not only the text prompts but also the normalization layers in the visual branch.

Reviewer comment (Theory):
The theoretical derivation of Eq. (3) & (4) could be more explicit and detailed. The current version appears to be intuitive and lack thorough theoretical analysis. Eq. (3) is proposed just to satisfy the rule that minimize the entropy ID instances and maximization the entropy of OOD samples. However, no theoretical guarantee is provided so that Eq. (3) & (4) would always satisfy the above principle. The explanation is also missing of how Eq. (3) & (4) would be more suitable than a simple stepwise function, e.g. \(L_{ID}=H(p(x))\) and \(L_{OOD}=-H(p(x))\), and the determination of OOD samples follows the common practice as introduced in Sec. 3.1.

Author response:
> **[Q3]**. How do Eq. (3) and Eq. (4) theoretically satisfy the principle "minimize the entropy of ID instances and maximize the entropy of OOD samples"?

**[A3]**. As explained above, when $w(x)\to 1$ for unlabeled ID samples and $w(x)\to 0$ for unlabeled OOD samples, Eq. (3) becomes the optimal objective that minimizes the entropy of ID instances and maximization the entropy of OOD samples. In the context of the unsupervised fine-tuning problem, where binary labels ("ID" or "OOD") are not available, we suggest employing the maximum softmax probability as an approximation for the ideal weight, as outlined in Eq. (3).

As the estimated weight $w(x)$ may not always be accurate — for example, no OOD samples exist in the unlabeled data under the closed-set and partial-set category shifts - maximizing the entropy can be harmful. Intuitively, we could maximize the entropy of mean prediction over OOD samples instead of the average entropy of individual OOD samples, leveraging the inequality $H(\mathbb{E}(p(x))) \geq \mathbb{E}(H(p(x)))$. Honestly, optimizing Eq. (3) and Eq. (4) does not theoretically guarantee the principle in the challenging unsupervised fine-tuning problem, but we believe the proposed objective in Eq. (4) establishes a simple yet strong baseline, as evidenced in extensive experiments.

Reviewer comment (Experiments):
As for the scope of application of the proposed method, it appears to be a general OOD method that can be also applied to traditional classification networks. I wonder why this method is applied to only CLIP method instead of extending it to other pretrained backbones.

Author response:
> **[Q5]**. The proposed method appears to be a general OOD method that can be also applied to traditional classification networks. I wonder why this method is applied to only CLIP method instead of extending it to other pre-trained backbones.

**[A5]**. Indeed, our objective in Eq. (4) is adaptable to traditional classification networks as well. The reason why we focus on CLIP lies in its flexibility, as a pre-trained model can be easily acquired based on class names without the need for labeled data, rendering it a versatile framework.

Regarding the traditional classification networks, we employ the ResNet-50 backbone with the [backbone - bottleneck layer - classifier layer] architecture to train the source model. During unsupervised fine-tuning in the target domain, we only update the parameters of the bottleneck layer (parameter-efficient), the results on two different tasks are shown below. On both tasks, UEO significantly outperforms other methods in terms of both ACC and AUC metrics.

| OF (D->A) | ACC (C) | AUC (C) | ACC (P) | AUC (P) | ACC (O) | AUC (O) | ACC (OP) | AUC (OP) | ACC (Avg.) | AUC (Avg.) |
|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|
| source only   | 65.2 | 71.8 | 65.2 | 71.8 | 65.2 | 71.8 | 65.2 | 71.8 | 65.2 | 71.8 |
| UPL        | 69.7 | 73.8 | 67.7 | 72.2 | 69.9 | 74.0 | 68.1 | 70.0 | 68.8 | 72.5 |
| POUF       | 69.9 | 75.7 | 68.9 | 73.7 | 70.1 | 74.2 | 68.5 | 71.7 | 69.4 | 73.8 |
| DANCE      | 65.0 | 72.5 | 66.9 | 73.5 | 65.8 | 73.2 | 64.8 | 71.8 | 65.6 | 72.7 |
| EntMin     | 64.3 | 72.0 | 64.9 | 72.8 | 63.2 | 69.8 | 63.9 | 70.5 | 64.1 | 71.3 |
| InfoMax    | 71.8 | 77.1 | 70.1 | 74.5 | 72.2 | 74.2 | 69.4 | 70.8 | 70.9 | 74.2 |
| UEO        | 71.6 | 75.5 | 71.1 | 74.9 | 71.6 | 75.0 | 70.2 | 74.1 | 71.1 | 74.9 |

| OH (Pr->Ar) | ACC (C) | AUC (C) | ACC (P) | AUC (P) | ACC (O) | AUC (O) | ACC (OP) | AUC (OP) | ACC (Avg.) | AUC (Avg.) |
|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|
| source only   | 54.6 | 64.5 | 54.5 | 64.5 | 54.6 | 64.5 | 54.5 | 64.5 | 54.6 | 64.5 |
| UPL        | 57.0 | 65.3 | 55.6 | 65.8 | 56.2 | 65.7 | 55.4 | 64.9 | 56.1 | 65.4 |
| POUF       | 57.1 | 64.6 | 56.1 | 64.6 | 57.2 | 63.6 | 56.4 | 63.4 | 56.7 | 64.0 |
| DANCE      | 53.9 | 65.2 | 54.2 | 65.0 | 53.2 | 62.7 | 54.0 | 64.7 | 53.8 | 64.4 |
| EntMin     | 52.5 | 64.3 | 53.2 | 64.4 | 51.5 | 62.6 | 52.1 | 62.4 | 52.3 | 63.4 |
| InfoMax    | 57.9 | 65.4 | 56.9 | 65.5 | 57.2 | 63.8 | 57.1 | 63.6 | 57.3 | 64.5 |
| UEO        | 58.3 | 66.3 | 58.2 | 66.2 | 57.9 | 64.9 | 58.2 | 64.7 | 58.1 | 65.5 |

Reviewer comment (Theory):
From my point of view, Eq. (3) can also be viewed as an implicit threshold strategy to determine OOD samples. Specifically, assume the max softmax probability \(w\) follows a uniform assumption \(w\sim \mathcal U(\frac{1}{C},1)\), where \(C\) denotes the total number of ID classes. The expectation \(\mathbb E(w)=\frac{1}{2}(1-\frac{1}{C^2})\) and \(\mathbb E(\frac 1 w)=\log(C)\). Therefore, \(\tilde w(x) - \tilde \Phi(w(x))\approx \frac{w}{\mathcal B_t \mathbb E(w)} - \frac{1/w}{\mathcal B_t \mathbb E(1/w)}\), and the threshold for determining whether a sample is OOD now becomes \(\lambda=\frac{1}{2} (1-\frac{1}{c^2})\log(C)\). In other words, Eq. (3) could be also one implicit form of thresholding strategy. I think the author should state the explicit benefit brought by the unified form as in Eq. (3) and (4) compared to a hard thresholding one. For example, we can observe in the form of \(\lambda\) that \(\lambda\) increases with the number of classes \(C\), yet I could not understand the rationale of this property.

Author response:
> **[Q4]**. The explanation is also missing of how Eq. (3) & (4) would be more suitable than a simple stepwise function, e.g. $L_{ID}=H(p(x))$ and $L_{OOD}=-H(p(x))$, and the determination of OOD samples follows the common practice as introduced in Sec. 3.1.

**[A4]**. Thanks for the advice. As written in the common practice in Sec. 3.1, $\lambda$ is chosen so that a high fraction of ID data (e.g., 90%) is above the threshold. Given the absence of explicit ID data in the unsupervised fine-tuning problem, we exclude it as a baseline method, denoted as **EO-ID**, in our experiments.

Following the interesting view from the reviewer, we assume $w\sim \mathcal{U}(\frac{1}{C},1)$, then obtain $\mathbb{E}(w)=\frac{1-\frac{1}{C^2}}{2}\times\frac{1}{1-\frac{1}{C}}$ and $\mathbb{E}(\frac{1}{w})=log(C)\times\frac{1}{1-\frac{1}{C}}$. To discover the sample whose entropy is minimized in Eq. (3), we have $\hat{w}(x) - \hat{\Phi}(w(x))>0 \Rightarrow \frac{w}{\mathbb{E}(w)} > \frac{1/w}{\mathbb{E}(\frac{1}{w})} \Rightarrow w^2\times \mathbb{E}(\frac{1}{w})> \mathbb{E}(w) \Rightarrow w > \sqrt{\frac{1-\frac{1}{C^2}}{2log(C)}}$. By the way, the threshold $\lambda=\sqrt{\frac{1-\frac{1}{C^2}}{2ln(C)}}$ sounds reasonable since it decreases when the number of classes ($C$) increases. We refer to the hard thresholding variant utilizing this implicit threshold as **EO-H**.

Here we conduct a simple comparison on the first domain (Ar) of the OfficeHome dataset and show the results across different category shifts below (**EO** denotes the objective in Eq. (3)).

| Metrics (%) | ACC (C) | AUC (C) | ACC (P) | AUC (P) | ACC (O) | AUC (O) | ACC (OP) | AUC (OP) | ACC (Avg.) | AUC (Avg.) |
|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|---------|
| CLIP    | 70.1 | 74.9 | 70.1 | 74.9 | 70.2 | 74.9 | 70.1 | 74.9 | 70.1 | 74.9 |
| EO-ID   | 69.9 | 75.7 | 69.8 | 75.6 | 70.4 | 76.3 | 70.4 | 76.3 | 70.1 | 76.0 |
| EO-H    | 68.8 | 74.7 | 68.8 | 74.4 | 69.6 | 76.2 | 70.1 | 76.0 | 69.3 | 75.3 |
| EO      | 68.9 | 74.9 | 68.6 | 74.7 | 69.6 | 75.8 | 69.5 | 75.7 | 69.1 | 75.3 |
| UEO     | 72.3 | 75.1 | 72.1 | 75.7 | 73.2 | 74.0 | 72.3 | 74.9 | 72.5 | 74.9 |

[(C): closed-set, (P): partial-set, (O): open-set, (OP): open-partial]

As can be seen from the table above, UEO consistently outperforms all three thresholding methods in terms of ACC while being inferior to them in terms of AUC under open-set and open-partial shifts. Notably, **the thresholding technique proves harmful for closed-set and partial-set shifts** where no OOD samples exist in the unlabeled data. In summary, UEO demonstrates superior suitability compared to these thresholding variants.