Paper: Can We Evaluate Domain Adaptation Models Without Target-Domain Labels? (ICLR-2024)

Reviewer comment (Experiments):
Ablation study of Table 3 is weak, authors should provide the same one like Fig. 2 to evaluation the effectiveness of each term.

Author response:
**Q2: The authors should supplement the ablation study of Table 3.**

**Answer**: We appreciate the suggestion and supplement two new UDA methods (CST [3] and AaD [4]) on a different dataset (VisDA-17) for the ablation study. As shown in the table below, the proposed transfer score selects the best model of 83.8% for CST and 85.9% for AaD. The results align with those of DomainNet. We have added the results to the revised manuscript in Table 3.

| Uniformity   | Hopkins      | MI           | CST (VisDA-17) | AaD (VisDA-17) | MCC (DomainNet) |
|--------------|--------------|--------------|----------------|----------------|-----------------|
| $\checkmark$ |              |              | 81.8           | 84.9           | 35.8            |
|              | $\checkmark$ |              | 82.2           | 84.9           | 35.8            |
|              |              | $\checkmark$ | 82.7           | 84.9           | 35.9            |
| $\checkmark$ | $\checkmark$ |              | **83.8**           | 85.3           | 36.0            |
|              | $\checkmark$ | $\checkmark$ | 83.3           | 85.8           | 36.5            |
| $\checkmark$ |              | $\checkmark$ | 83.3           | 85.6           | 36.5            |
| $\checkmark$ | $\checkmark$ | $\checkmark$ | **83.8**       | **85.9**       | **37.3**        |

Reviewer comment (Theory):
The representation of Equation 1 is incorrect, especially i*(i-1) and the use of i.

Author response:
**Q3: The Eq.1 is incorrect regarding the use of $i$.**

**Answer**: We appreciate the careful check of the reviewer regarding the wrong formulas in Eq.(1). We have revised it in the manuscript.

Reviewer comment (Experiments):
Authors should provide the complete results of all tasks from the different datasets * different UDA methods * baseline methods and proposed score, like Fig.3. The experimental results in the paper seem a bit sparse.

Author response:
**Q4: Authors should provide the complete results of other tasks.**

**Answer**: As suggested by the reviewer, we have added experiments on all other tasks of Office-31 and Office-Home. For Offce-31, we use all 4 tasks except the W$\to$D and D$\to$W as their transfer accuracy is over 99% which does not show the domain gap for UDA. For Office-Home, we supplement all 12 tasks. The results of Office-31 and Office-Home are shown as follows. In most cases, the transfer score can help select the best-performing model across the candidates on all tasks. Due to the page limit, these results have been summarized in Fig. 8 in the appendix of the revised manuscript.

| Method |      A$\to$D     |      A$\to$W     |      D$\to$A     |      W$\to$A     |
|:------:|:----------------:|:----------------:|:----------------:|:----------------:|
|   DAN  |   77.7 / 1.542   |   81.6 / 1.542   |   55.5 / 1.478   |   56.4 / 1.492   |
|  DANN  |   81.9 / 1.587   |   87.7 / 1.592   |   73.3 / 1.600   |   72.3 / 1.592   |
|  CDAN  |   89.4 / 1.590   | **93.7 / 1.635** |   73.4 / 1.605   |   73.4 / 1.632   |
|   MDD  |   93.2 / 1.596   |   92.7 / 1.604   |   75.5 / 1.607   |   73.5 / 1.606   |
|   MCC  |   93.6 / 1.617   |   92.8 / 1.632   | **75.5 / 1.660** | **74.1 / 1.665** |
|   FixMatch   | **95.8 / 1.706** |   88.1 / 1.618   |   68.0 / 1.531   |   65.0 / 1.510   |

| Method   | A$\to$P          | A$\to$R          | C$\to$A          | C$\to$R          | P$\to$A          | P$\to$C          | R$\to$C          | R$\to$P          |
|----------|------------------|------------------|------------------|------------------|------------------|------------------|------------------|------------------|
| DAN      | 66.9 / 1.400     | 73.2 / 1.410     | 56.5 / 1.285     | 66 / 1.356       | 50.8 / 1.300     | 39.0 / 1.293     | 47.6 / 1.332     | 78.5 / 1.441     |
| DANN     | 60.1 / 1.367     | 73.2 / 1.45      | 56.9 / 1.410     | 67.9 / 1.433     | 58.4 / 1.439     | 53.8 / 1.460     | 60.8 / 1.469     | 80.2 / 1.500     |
| CDAN     | 71.1 / 1.500     | 76.8 / 1.498     | 60.6 / 1.429     | 72.4 / 1.458     | 61.1 / 1.455     | 54.4 / 1.470     | 58.7 / 1.449     | 83.0 / 1.515     |
| MDD      | 75.4 / 1.541     | 78.8 / 1.539     | 61.9 / 1.474     | 73.8 / 1.509     | 61.8 / 1.462     | 54.7 / 1.502     | 59.7 / 1.517     | 84.3 / 1.548     |
| MCC      | **79.3 / 1.557** | **82.6 / 1.544** | **67.1 / 1.487** | **77.6 / 1.515** | **66.5 / 1.488** | 54.7 / 1.518     | 61.6 / 1.529     | **85.5 / 1.550** |
| FixMatch | 73.9 / 1.536     | 80.2 / 1.542     | 64.8 / 1.451     | 70.2 /1.493      | 66.2 / 1.471     | **57.6 / 1.549** | **64.0 / 1.541** | 84.9 / 1.549     |

Reviewer comment (Experiments):
What if the Transfer Score used as a optimization function for UDA task? I am curious about this.

Author response:
**Q5: What if the transfer score is used as a learning objective for UDA task? I am curious about this.**

**Answer**: We appreciate the interesting suggestion and add the experiment to explore the effectiveness of the three components of the transfer score. The experiments are conducted on Office-31 (A$\to$W) and Office-Home (Ar$\to$Cl) based on ResNet50. We add the three parts of transfer score as an independent regularizer on the target domain. As shown in the table below, it is shown that every component brings some improvement compared to the source-only model. The total transfer score even brings significant improvement by 24.1% for A$\to$W and 12.6% for A$\to$W. The vanilla hopkins statistic is calculated at the batch level so it can be enhanced further as a learning objective for UDA. We think this is an explorable direction as an independent work in the future. The table and analytics have been added to the revised manuscript in Appendix A.10.

|                        | A$\to$W | Ar$\to$Cl |
|------------------------|:-------:|:---------:|
| Source-only            |   68.4  |    34.9   |
| Uniformity             |   75.1  |    41.8   |
| Hopkins statistic      |   75.6  |    41.8   |
| Mutual information     |    92   |    45.6   |
| Transfer Score (Total) |   **92.5**  |    **47.6**   |

Reviewer comment (Evaluation):
Mutual information in Eq.4 is used as a term in the Transfer Score, if some methods (e.g. SHOT) minimize this term during adaptation, will it directly cause mutual information term to become invalid?

Author response:
**Q6+Q7: Some methods, e.g., SHOT, use the mutual information as an objective. Will it cause the proposed method become invalid? Can authors also provide some results on source-free domain adaptation (SFDA) methods (e.g., SHOT)?**

**Answer**: Our method still works when Eq.4 (mutual information) is used as a learning objective. As we know, mutual information regularizer is used in SFDA methods (e.g., SHOT and AaD which uses a similar regularizer of Eq.4). Despite this, our proposed transfer score consists of three components, which is designed to avoid the single evaluation metric in previous unsupervised validation work (SND or C-Entropy). With three components, even though the mutual information is optimized by training, the other two metrics still work well. As suggested by the reviewer, we add experiments using two novel source-free UDA methods, SHOT (TPAMI’21) [2] and AaD (NeurIPS’22) [4], on Office-Home and VisDA-17. The results are shown as follows:
- The first table shows the accuracy and the transfer score on Office-Home. The AaD achieves the best results on Office-Home with the highest transfer score. 
- The second table shows the improvement of epoch selection by transfer score on VisDA-17. Our method effectively selects a better epoch of the model for both methods, improving SHOT and AaD by 0.5% and 2.5%, respectively.

| Method |   Ar$\to$Cl  |   Cl$\to$Pr  |   Pr$\to$Rw  |   Rw$\to$Ar  |
|:------:|:------------:|:------------:|:------------:|:------------:|
|  SHOT  | 56.7 / 1.532 | 74.0 / 1.522 | 82.1 / 1.570 | 72.3 / 1.544 |
|   AaD  | 59.1 / 1.700 | 80.8 / 1.706 | 83.0 / 1.696 | 72.3 / 1.551 |

| Method | Last Epoch | Ours | Imp. $\uparrow$ |
|:------:|:----------:|:----:|:---------------:|
|  SHOT  |    77.5    | 78.0 |       0.5       |
|   AaD  |    83.4    | 85.9 |       2.5       |

Therefore, the results show that the transfer score still works on source-free domain adaptation methods that leverage mutual information as a regularizer. We have added the results to Fig.8. As suggested by other reviewers, we also supplement the results on cross-domain semantic segmentation [1] and CAN [5] to the revised manuscript, which further demonstrates the effectiveness of transfer score on more tasks and totally 11 UDA baseline methods.

Reviewer comment (Experiments):
Because there is no source information used in Transfer Score, can authors provide the results on Source-Free Domain Adaptation methods?

Author response:
**Q6+Q7: Some methods, e.g., SHOT, use the mutual information as an objective. Will it cause the proposed method become invalid? Can authors also provide some results on source-free domain adaptation (SFDA) methods (e.g., SHOT)?**

**Answer**: Our method still works when Eq.4 (mutual information) is used as a learning objective. As we know, mutual information regularizer is used in SFDA methods (e.g., SHOT and AaD which uses a similar regularizer of Eq.4). Despite this, our proposed transfer score consists of three components, which is designed to avoid the single evaluation metric in previous unsupervised validation work (SND or C-Entropy). With three components, even though the mutual information is optimized by training, the other two metrics still work well. As suggested by the reviewer, we add experiments using two novel source-free UDA methods, SHOT (TPAMI’21) [2] and AaD (NeurIPS’22) [4], on Office-Home and VisDA-17. The results are shown as follows:
- The first table shows the accuracy and the transfer score on Office-Home. The AaD achieves the best results on Office-Home with the highest transfer score. 
- The second table shows the improvement of epoch selection by transfer score on VisDA-17. Our method effectively selects a better epoch of the model for both methods, improving SHOT and AaD by 0.5% and 2.5%, respectively.

| Method |   Ar$\to$Cl  |   Cl$\to$Pr  |   Pr$\to$Rw  |   Rw$\to$Ar  |
|:------:|:------------:|:------------:|:------------:|:------------:|
|  SHOT  | 56.7 / 1.532 | 74.0 / 1.522 | 82.1 / 1.570 | 72.3 / 1.544 |
|   AaD  | 59.1 / 1.700 | 80.8 / 1.706 | 83.0 / 1.696 | 72.3 / 1.551 |

| Method | Last Epoch | Ours | Imp. $\uparrow$ |
|:------:|:----------:|:----:|:---------------:|
|  SHOT  |    77.5    | 78.0 |       0.5       |
|   AaD  |    83.4    | 85.9 |       2.5       |

Therefore, the results show that the transfer score still works on source-free domain adaptation methods that leverage mutual information as a regularizer. We have added the results to Fig.8. As suggested by other reviewers, we also supplement the results on cross-domain semantic segmentation [1] and CAN [5] to the revised manuscript, which further demonstrates the effectiveness of transfer score on more tasks and totally 11 UDA baseline methods.

Reviewer comment (Experiments):
In the experiments, only six UDA methods were evaluated. In task 2, there are only five candidate hyperparameters. The number is too small to illustrate the effectiveness of the proposed method.

Author response:
We would like to summarize the difference between our metric and the existing unsupervised evaluation method (SND) in the table below. We have supplemented the experiments and the differences in the revised manuscript.

|   | Ours   | SND    |
|----------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|
| Task | Model comparison Hyper-parameter tuning Epoch selection | Epoch selection Hyper-parameter tuning   |
| Method  | Transfer score is a three-fold metric, including uniformity of model weights, the mutual information of features, and the clustering tendency.  | SND uses a single metric, the density of implicit local neighborhoods, which describes the clustering tendency. |
| Baseline UDA methods | 11 UDA methods (including 4 new methods in the rebuttal): 1. Adversarial: CDAN, DANN, MDD 2. Moment matching: DAN, CAN 3. Reweighing-based: MCC 4. Self-training: FixMatch, CST, SHOT, AaD 5. Norm-based: SAFN | 4 UDA methods: 1. Adversarial: CDAN 2. Reweighing-based: MCC 3. Self-training: NC, PL   |

**Q2: Only six UDA methods are evaluated and five candidate hyperparameters are provided. The number of these candidates should be increased.**

**Answer**: We appreciate the suggestion and add extensive experiments to address this issue.  

Firstly, we supplement four new UDA methods including SHOT [2] (TPAMI-21), CST [3] (NeurIPS-21), AaD [4] (NeurIPS-22), and CAN [5] (CVPR-19). The experiments are conducted on Office-Home and VisDA-17 and we use their open-source codes as the backbones. The results have been shown as follows:
- The first table shows the *accuracy* and *transfer score*. Our metric successfully selects the best-performing UDA method with the highest transfer score for these tasks. 
- The second table demonstrates that using transfer score can help select a better epoch on VisDA-17, improving SHOT, CAN, CST and AaD by 0.5%, 0.4%, 11.8%, and 2.5%, respectively. These comparisons have been summarized and updated in Fig.2 in the revised manuscript and the appendix.

|  Method  |     Ar$\to$Cl    |     Cl$\to$Pr    |     Pr$\to$Rw    |     Rw$\to$Ar    |
|:--------:|:----------------:|:----------------:|:----------------:|:----------------:|
|    DAN   |   44.2 / 1.321   |   63.7 / 1.363   |   73.2 / 1.400   |   66.4 / 1.343   |
|  CAN [5] |   50.1 / 1.390   |   70.8 / 1.381   |   76.3 / 1.422   |   68.5 / 1.412   |
| SHOT [2] |   56.7 / 1.532   |   74.0 / 1.522   |   82.1 / 1.570   |   72.3 / 1.544   |
|  CST [3] |   57.8 / 1.570   |   75.9 / 1.580   |    82.5/ 1.585   | **75.4 / 1.638** |
|  AaD [4] | **59.1 / 1.700** | **80.8 / 1.706** | **83.0 / 1.696** |   72.3 / 1.551   |

|  Method  | Last | Ours | Imp. $\uparrow$ |
|:--------:|:----:|:----:|:---------------:|
| SHOT [2] | 77.5 | 78.0 |       0.5       |
|  CAN [5] | 86.4 | 86.8 |       0.4       |
|  CST [3] | 72.0 | 83.8 |       11.8      |
|  AaD [4] | 83.4 | 85.9 |       2.5       |

In summary, we demonstrate that our method works effectively on four more UDA methods, two of which are source-free domain adaptation methods. Compared to the SOTA method SND, we have 11 UDA baselines to prove the effectiveness while SND only uses 4 UDA baselines.

For the hyper-parameter candidate, we have enriched the candidate to [1, 2, 4, 6, 8, 10, 12, 14, 16] and the results have been shown in the first table in Q1. We demonstrate that our method can effectively select a better hyperparameter than SND.

Reviewer comment (Experiments):
The UDA datasets employed in the experiments are not comprehensive. On office-home, only four tasks were selected, and the situation was similar for DomainNet and Office. How do you select these tasks?

Author response:
**Q3: How do you select the transfer tasks on the datasets?**

**Answer**: Following SND, we select some tasks that can contain all domains in Office-31 and Office-Home. The two tasks in DomainNet are selected as the evaluation of imbalanced long-tailed tasks. **As suggested by the reviewer, we realize that this is not comprehensive. Therefore we have added experiments on all other tasks of Office-31 and Office-Home.** For Offce-31, we use all 4 tasks except the W$\to$D and D$\to$W as their transfer accuracy is over 99% which does not show the domain gap for UDA. For Office-Home, we supplement all 12 tasks. The results of Office-31 and Office-Home are shown as follows (format: accuracy / transfer score). On Office-31, our method selects FixMatch, CDAN, and MCC for the best-performing models, which is very accurate. Our method also works effectively on all 12 tasks on Office-Home (whose table is displayed in the next paragraph). The results have been summarized in Fig. 8 in the appendix of the revised manuscript.

| Method |      A$\to$D     |      A$\to$W     |      D$\to$A     |      W$\to$A     |
|:------:|:----------------:|:----------------:|:----------------:|:----------------:|
|   DAN  |   77.7 / 1.542   |   81.6 / 1.542   |   55.5 / 1.478   |   56.4 / 1.492   |
|  DANN  |   81.9 / 1.587   |   87.7 / 1.592   |   73.3 / 1.600   |   72.3 / 1.592   |
|  CDAN  |   89.4 / 1.590   | **93.7 / 1.635** |   73.4 / 1.605   |   73.4 / 1.632   |
|   MDD  |   93.2 / 1.596   |   92.7 / 1.604   |   75.5 / 1.607   |   73.5 / 1.606   |
|   MCC  |   93.6 / 1.617   |   92.8 / 1.632   | **75.5 / 1.660** | **74.1 / 1.665** |
|   FixMatch   | **95.8 / 1.706** |   88.1 / 1.618   |   68.0 / 1.531   |   65.0 / 1.510   |


Results on other 8 tasks on Office-Home:

| Method   | A$\to$P          | A$\to$R          | C$\to$A          | C$\to$R          | P$\to$A          | P$\to$C          | R$\to$C          | R$\to$P          |
|----------|------------------|------------------|------------------|------------------|------------------|------------------|------------------|------------------|
| DAN      | 66.9 / 1.400     | 73.2 / 1.410     | 56.5 / 1.285     | 66 / 1.356       | 50.8 / 1.300     | 39.0 / 1.293     | 47.6 / 1.332     | 78.5 / 1.441     |
| DANN     | 60.1 / 1.367     | 73.2 / 1.45      | 56.9 / 1.410     | 67.9 / 1.433     | 58.4 / 1.439     | 53.8 / 1.460     | 60.8 / 1.469     | 80.2 / 1.500     |
| CDAN     | 71.1 / 1.500     | 76.8 / 1.498     | 60.6 / 1.429     | 72.4 / 1.458     | 61.1 / 1.455     | 54.4 / 1.470     | 58.7 / 1.449     | 83.0 / 1.515     |
| MDD      | 75.4 / 1.541     | 78.8 / 1.539     | 61.9 / 1.474     | 73.8 / 1.509     | 61.8 / 1.462     | 54.7 / 1.502     | 59.7 / 1.517     | 84.3 / 1.548     |
| MCC      | **79.3 / 1.557** | **82.6 / 1.544** | **67.1 / 1.487** | **77.6 / 1.515** | **66.5 / 1.488** | 54.7 / 1.518     | 61.6 / 1.529     | **85.5 / 1.550** |
| FixMatch | 73.9 / 1.536     | 80.2 / 1.542     | 64.8 / 1.451     | 70.2 /1.493      | 66.2 / 1.471     | **57.6 / 1.549** | **64.0 / 1.541** | 84.9 / 1.549     |

Reviewer comment (Novelty):
The authors said, "TS is the **first** metric to perform simultaneous model comparison and selection without target-domain labels." As far as I know, the three tasks in experiments are actually unsupervised validation problems, and this is an existing research field. Besides, some out-of-distribution generalization prediction methods also solve the model selection problem without labels, for example:  
> K-Means Clustering Based Feature Consistency Alignment for Label-Free Model Evaluation.  
> Predicting Out-of-Distribution Error with Confidence Optimal Transport.  
> Leveraging Unlabeled Data to Predict Out-of-Distribution Performance.  
> ...

Author response:
**Q4:  The three tasks in experiments are actually unsupervised validation (UV) problems, and this is an existing research field. Besides, some out-of-distribution generalization prediction methods also solve the model selection problem without labels.**

**Answer**:
We apologize for not clearly illustrating the difference between existing unsupervised validation methods and ours. We would like to elaborate on the difference from three perspectives:
1. Existing UV methods (SND, C-entropy) aim at selecting hyper-parameters or epochs for the same UDA method, while our metric can conduct a UDA method comparison (Task 1) for the first time. We demonstrate in Q1 that other metrics cannot achieve the model comparison task. Therefore, we claim that our method is the first work to “perform simultaneous model comparison and selection without target-domain labels”. 
2. For the Task 2&3, we demonstrate our method outperforms existing methods on more datasets using more categories of UDA methods. The reason is that our metric considers multiple aspects of the model while others use a single metric. For example, SND only uses the clustering characteristics and C-Entropy only uses the prediction uncertainty of samples.
3. We have already considered the UV methods in the out-of-distribution (OOD) research community. These methods are mostly based on sample-level uncertainty while ours rely on model-level and manifold-level metrics. The experiments have been made between ours and ATC (ICLR’22) [6] in Table 2. Our transfer score outperforms ATC on Office-Home and VisDA-17 based on two novel UDA baselines.

**References**

[1] Tune it the right way: Unsupervised validation of domain adaptation via soft neighborhood density. ICCV 2021.

[2] Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer, T-PAMI 2021

[3] Cycle self-training for domain adaptation, NeurIPS 2021

[4] Yang S, Jui S, van de Weijer J. Attracting and dispersing: A simple approach for source-free domain adaptation. NeurIPS 2022

[5] Kang G, Jiang L, Yang Y, et al. Contrastive adaptation network for unsupervised domain adaptation, CVPR 2019

[6] Leveraging unlabeled data to predict out-of-distribution performance. ICLR 2022.

[7] Minimal-entropy correlation alignment for unsupervised deep domain adaptation. ICLR 2018.

Reviewer comment (Experiments):
As the authors stated, “prevailing UDA methods relying on adversarial training and self-training could lead to model degeneration and negative transfer”. How can we prove this viewpoint? Can the proposed metric be applied to these types of methods for performance improvement in regular UDA settings?

Author response:
**Q1: How can we prove that “prevailing UDA methods could lead to model degeneration and negative transfer”? Can the proposed metric be applied to them for performance improvement in regular UDA settings?**

**Answer**: There are two probable situations when UDA methods lead to negative transfer: 
1. inappropriate hyper-parameters: Since we do not have labeled target-domain samples for hyper-parameter tuning, the inappropriate hyper-parameters could lead to deteriorating convergence. As shown in Fig.3, for three UDA methods (DANN, SAFN, MDD), wrong hyper-parameters lead to even worse performances than the source-only model. This is usually caused by a greater weight of the transfer loss that dominates the training.
2. Overfitting to source domain: For large-scale UDA datasets, it is observed that the model is prone to overfitting to the source domain after a few epochs of training, leading to negative transfer. As shown in Fig.4, we see the overfitting happens on VisDA-17 and DomainNet. The SAFN even drops from 41% to 25% on DomainNet.

The issue of negative transfer is also studied in some literature [7] that analyzes the reasons behind it. Our metric can effectively deal with the issues of negative transfer for regular UDA settings. In practice, as we do not have any target-domain labels, we usually use hyper-parameters and a fixed number of epochs sourced from the paper. However, the UDA method performs distinctly on different data, so the negative transfer may happen due to the aforementioned reasons. Our method is designed to work effectively in regular UDA settings. The task 2 and 3 in our work aim to overcome the issue by unsupervised validation on the target domain. The results in Table 1 and Fig.3 show the improvement of various UDA methods on different public datasets.

Reviewer comment (Evaluation):
Although the proposed metric is intuitive, it is difficult to validate that it is definitely correct for evaluating a UDA model. In fact, various existing UDA approaches also adopt “Transfer Scores” for guiding the model learning process, such as [a]. A comparison between these metrics should be conducted.

Author response:
**Q2: Some existing UDA methods also adopt a metric for guiding the model learning [1]. A comparison should be made.**

**Answer**: We appreciate the suggestion and read the paper carefully. This paper proposes an active domain adaptation approach (CNTGE) based on a transferability metric. We would like to illustrate their differences regarding the tasks, method, and experiments.

|  | Transfer Score | CNTGE |
|---|---|---|
| Task | Unsupervised validation of UDA methods | Active domain adaptation |
| Method | Measure the uniformity of model weights, the mutual information of features, and the clustering tendency. | The metric proposed by CNTGE is used to select samples by the sample-level uncertainty of prediction. |
| Significance | The transfer score is a generic metric to perform model comparison and selection for UDA methods. | The CNTGE is a novel active domain adaptation approach that achieves good results on benchmarks. |

In the table, we see that the proposed metric is different from the sample-level transfer score in CNTGE. The transfer score proposed by CNTGE aims at selecting uncertain samples for active domain adaptation. It is a simple but effective metric for sample selection during UDA training, while our transfer score is a generic metric for unsupervised validation. Our metric measures the model-level transferability of the uniformity of model weights, the mutual information of features, and the clustering tendency. We have supplemented the contents in the manuscript.

Reviewer comment (Experiments):
In figure 1, the correlation analysis is conducted only in one UDA task (Office31 A->W) and one UDA model (DANN). More comprehensive analysis and more comparison (with more metrics) should be conducted.

Author response:
**Q3: Figure 1 is only conducted in one UDA task and model. More analysis should be conducted.**

**Answer**: We appreciate this suggestion. The Fig.1 is originally regarded as an intuitive figure that shows the difference between existing metrics and our metric by a preliminary experiment. As suggested by the reviewer, we supplement the experiment by comparing our metric with two advanced unsupervised validation metrics, C-Entropy [8] and SND [9], on Office-31 (A$\to$W) and VisDA-17. Six figures have been added to the revised manuscript (*in the appendix, Fig.11 and Fig.12*). For convenience, we also provide the table below to show the **correlation coefficient** between the metrics and the model accuracy as follows. For both datasets, our approach shows the best correlation coefficients. Fig.12 shows that for large-scale datasets like VisDA-17, only our metric can reflect the overfitting issue while the other two metrics that keep increasing during training contradict the decreasing accuracies.

|  | C-Entropy | SND | Ours |
|---|---|---|---|
| Office-31  | 0.95 | 0.74 | 0.97 |
| VisDA-17 | 0.85 | 0.57 | 0.90 |

Reviewer comment (Theory):
In Definition 2, the design of the Score seems handcrafted. How can we ensure this formulation is optimal? Why use this design? The reasons should be analyzed.

Author response:
**Q4: In definition 2, how the authors design the score should be justified.**

**Answer**: We appreciate the suggestions and apologize for the lack of justification. In definition 2, we aim to use a larger transfer score to indicate better transferability. To this end, as the greater uniformity indicates a larger bias and lower transferability, we use the negative uniformity. In contrast, we use the Hopkins statistic and the absolute value of mutual information as they are positively correlated to the transferability. The mutual information is especially normalized because it does not have a fixed range as the uniformity and Hopkins statistic does. We have included the justification of the design in the revised manuscript.

Reviewer comment (Experiments):
Some SOTA UDA approaches are not employed for experimental analysis [b-c].

Author response:
**Q5: Some SOTA UDA approaches should be employed for experiments.**

**Answer**: As suggested by the reviewer, we supplement four new UDA methods including SHOT [2] (TPAMI-21), CST [3] (NeurIPS-21), AAD [4] (NeurIPS-22), and CAN [5] (CVPR-19). The experiments are conducted on Office-Home and VisDA-17 and we use their open-source codes as the backbones. The results have been shown as follows:
- The first table shows the **accuracy / transfer score** on Office-Home. Our metric successfully indicates the best-performing method with the highest transfer score for these tasks. 
- The second table demonstrates that using transfer score can help select a better epoch on VisDA-17, improving SHOT, CAN, CST, and AaD by 0.5%, 0.4%, 11.8%, and 2.5%, respectively. These comparisons have been summarized and updated in *Fig.2 in the revised manuscript and the appendix*.

| Method | Ar$\to$Cl | Cl$\to$Pr | Pr$\to$Rw | Rw$\to$Ar |
|:---:|:---:|:---:|:---:|:---:|
| DAN | 44.2 / 1.321 | 63.7 / 1.363 | 73.2 / 1.400 | 66.4 / 1.343 |
| CAN [5] | 50.1 / 1.390 | 70.8 / 1.381 | 76.3 / 1.422 | 68.5 / 1.412 |
| SHOT [2] | 56.7 / 1.532 | 74.0 / 1.522 | 82.1 / 1.570 | 72.3 / 1.544 |
| CST [3] | 57.8 / 1.570 | 75.9 / 1.580 | 82.5/ 1.585 | **75.4 / 1.638** |
| AaD [4] | **59.1 / 1.700** | **80.8 / 1.706** | **83.0 / 1.696** | 72.3 / 1.551 |

| Method | Last | Ours | Imp. $\uparrow$ |
|:---:|:---:|:---:|:---:|
| SHOT [2] | 77.5 | 78.0 | 0.5 |
| CAN [5] | 86.4 | 86.8 | 0.4 |
| CST [3] | 72.0 | 83.8 | 11.8 |
| AaD [4] | 83.4 | 85.9 | 2.5 |

In summary, we demonstrate that our method works effectively on four more UDA methods, two of which are even source-free domain adaptation methods. Note that we cannot reproduce Fixbi [6] since its open-source codes do not release its specially pre-trained DANN model parameters, which have been an issue raised and unsolved in Github repository. To ensure fair comparison we therefore decided not to use Fixbi.

Reviewer comment (Evaluation):
From Table 1, why are the improvements of the last four methods on the DomainNet (p->c) task so small?

Author response:
**Q6: In Table 1, why are the improvements of the last four methods on DomainNet (p$\to$c) so small?**

**Answer**: In Table 1, the improvement of our method is caused by selecting a better epoch after training. Hence, the margin of improvement is limited by the range of varying accuracy during training using a specific UDA backbone. For example, on DomainNet (p->c), we find that DANN shown in Fig.4(e) has a decreasing accuracy from 40.2% to 34.5%, so when our metric selects the best epoch, the improvement is 5.7%. However, some methods like MDD and MCC show stable training curves w.r.t. the accuracy. Even though our method selects a better epoch, the improvement is marginal. The improvements are closely dependent on the specific training curve based on the datasets and the UDA model.

Reviewer comment (Evaluation):
“Transfer Score” uses clustering and class balance as the measurement criteria. However, the assumption of clustering and class balance usually may not always hold. Acutally, in many real world tasks, class imbanlance often exists. For example, in cross-domain semantic segmentation, a severe category imbalance is often present [1], which limits the application of this metric.

Author response:
**Q1: “Transfer Score” uses clustering and class balance as the measurement criteria. Does the transfer score apply to the imbalanced UDA task, such as the cross-domain classification and semantic segmentation?**

**Answer**: Yes, the proposed transfer score works effectively on imbalanced UDA tasks. As illustrated in Section 5.5, to validate the effectiveness of handling imbalanced datasets, we have conducted an experiment on DomainNet (two transfer tasks) where severe imbalanced class distribution exists as shown in Fig.5(a). Our method still shows good results as shown in Table 1, Fig.5(b) and Fig.6. According to the reviewer’s suggestion, to better evaluate the imbalanced situation, we add a new experiment on semantic segmentation. We use a classic cross-domain segmentation approach FDA [1] as a baseline on GTA-5 -> Cityscapes. The mIoU results are shown below. It is shown that our metric can choose a better epoch of the model with an average mIoU of 39.7% while the last epoch of the model only achieves 37.7%. This demonstrates that our metric still works effectively on the imbalanced cross-domain semantic segmentation task.

| Class | road | sidewalk | building | wall | fence | pole | light | sign | vegetation | terrain | sky | person | rider | car | truck | bus | train | motocycle | bicycle | Avg |
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| Vanilla | 79.3 | 27.4 | 76.3 | 23.6 | 24.3 | 25.1 | 28.5 | 18.4 | 80.4 | 32.3 | 71.6 | 53.0 | 14.1 | 75.1 | 24.2 | 30.1 | 7.6 | 15.1 | 12.6 | 37.7 |
| Ours | 80.7 | 29.5 | 80.4 | 30.1 | 23.6 | 28.8 | 28.3 | 15 | 80.8 | 32.1 | 79.1 | 55.5 | 11.0 | 79.8 | 33.8 | 38.9 | 5.2 | 15.6 | 8.6 | **39.7** |

Intuitively, the clustering metric (Hopkins statistic) measures the clustering tendency, which is irrelevant to class balance. The uniformity measures the classifier bias, i.e., how uniformly the feature space is divided by a classifier. It is directly calculated on model weights and does not measure the class balance. For UDA, the better uniformity of a classifier indicates better generalization ability. For example, if a classifier overfits the source domain, it should fail to perform well on the target domain with a different class distribution, and such a classifier has a low level of uniformity.

Reviewer comment (Evaluation):
Some methods [2,3] directly adopt Eq. 1 and Eq. 4 as optimization objectives. For these methods, the transfer score may be invalid.

Author response:
**Q2: Does the transfer score work in some methods (SHOT, AAD) that directly use Eq.1 and Eq.4 as learning objectives?**

**Answer**: Yes. Our method still works when Eq.4 (mutual information) is used as a learning objective. As mentioned by the reviewer, mutual information regularizer is very commonly used in source-free UDA methods (e.g., SHOT and AaD which uses a similar regularizer of Eq.4). Whereas, our proposed transfer score consists of three components, which is designed to avoid the single evaluation metric in previous unsupervised validation work. With three components, even though the mutual information is optimized by training, the other two metrics still work well. To prove it, we supplement the experiments using SHOT [2] and AaD [4] on Office-Home and VisDA-17. The results are shown as follows:
- The first table shows the *accuracy* and *transfer score* on Office-Home. The AaD achieves the best results on Office-Home with the highest transfer score. 
- The second table shows the improvement of epoch selection by transfer score on VisDA-17. Our method effectively selects a better epoch of the model for both methods, improving SHOT and AaD by 0.5% and 2.5%, respectively.
Therefore, the results show that the transfer score still works when Eq.4 is used as a learning objective. Eq.1 is proposed by us in this paper, and we have not found any methods using it as a learning objective.

| Method | Ar$\to$Cl | Cl$\to$Pr | Pr$\to$Rw | Rw$\to$Ar |
|:---:|:---:|:---:|:---:|:---:|
| SHOT [2] | 56.7 / 1.532 | 74.0 / 1.522 | 82.1 / 1.570 | 72.3 / 1.544 |
| AaD [4] | 59.1 / 1.700 | 80.8 / 1.706 | 83.0 / 1.696 | 72.3 / 1.551 |

| Method | Last | Ours | Imp. $\uparrow$ |
|:---:|:---:|:---:|:---:|
| SHOT [2] | 77.5 | 78.0 | 0.5 |
| AaD [4] | 83.4 | 85.9 | 2.5 |