From: ICLR.cc/2023/Conference/Program_Chairs
title: Paper Decision
decision: Reject
metareview:_summary,_strengths_and_weaknesses: Authors have designed a benchmark comparison for partial domain adaptation to evaluate the model selection strategies across two different real-world datasets. It also conducted comprehensive experiments to draw some conclusions.

While this is an important research area, it does not provide any solution to this problem. It is also recommended to use more datasets for more solid evaluation. In addition, the insights obtained is also not so significant. Authors also need to put more effort in the writing as it has typos and sometimes hard to follow.


justification_for_why_not_higher_score: While this is an important research area, it does not provide any solution to this problem. It is also recommended to use more datasets for more solid evaluation. In addition, the insights obtained is also not so significant. Authors also need to put more effort in the writing as it has typos and sometimes hard to follow.

justification_for_why_not_lower_score: N/A
summary_of_AC-reviewer_meeting: No need as reviewers have provided relatively low ratings

From: ICLR.cc/2023/Conference/Paper5333/Authors
title: End of discussion period is approaching
comment: Dear reviewers,

Thank you for taking the time to review our work and for your remarks. We appreciate all the positive comments. Nonetheless, there were different concerns raised. We have responded to each one of you individually and hope to have addressed them. As the end of the discussion period is approaching, we would really appreciate to know if our answers have convinced you.

Authors of submission "A Reproducible and Realistic Evaluation of Partial Domain Adaptation Methods"

From: ICLR.cc/2023/Conference/Paper5333/Authors
title: General Response
comment: Dear reviewers,

Thank you for taking the time to review our work and for your remarks. We appreciate all the positive comments: the significance and importance of this work to the community, highlighted by all reviewers, the extensive experiments performed (Reviewer PkyR, Reviewer unvz) and that our code is open-sourced for reproducibility (Reviewer ZmSX).

Nonetheless, there were different concerns raised. We have respond to each one of you individually and hope to have addressed them. We have also uploaded a revised version of our paper. For your convenience we highlighted all the changes in the pdf in blue.

Thank you again for your review. Please let us know if we need further clarifications before the deadline on November 18.

From: ICLR.cc/2023/Conference/Paper5333/Authors
title: Response Reviewer ZmSX
comment: We would like to thank the reviewer for the time taken on our manuscript and for acknowledging that our considered problem is of interest to the community.

>While the paper is well-written overall, some sentences are difficult to follow.

We have corrected all the sentences pointed out. However all of these constitute small typos that the reviewer was able to understand given the suggested corrections. Thus we kindly ask if the reviewer could point out any other specific sentences that remain difficult to follow.

>The paper is challenging to follow and understand.

We find this comment to be at odds with the comment cited just above that claims the paper to be "well-written overall" with only "some sentences [...] difficult to follow". We kindly ask if the reviewer could point out which specific parts of the paper remain hard to follow and understand. We would be happy to provide additional clarification.

> While the tackled problem of interest, a realistic evaluation of PDA methods, is original, the novelty of the work overall is still in question.

Regarding the novelty of our work, we kindly disagree with the reviewer. There are several apects in our work which are new and were not discussed in the litterature before. First of all, our work is the first to evaluate model selection criterion over several algorithms and datasets. These label free criterions were designed to lead to similar performance to using a target validation set. For instance, Saito et al (2021) claimed that SND leads to performance close to optimal in 80% of the cases. However our study invalidates this claim as for instance M-POT paired with SND has an average task accuracy on Office-Home of 46.92%, while with ORACLE the average accuracy is 77.31%. On the VisDA dataset, SND paired with SAFN, BA3US and AR also performs poorly in comparison to ORACLE. Such a drop in performance is significant and points towards PDA methods being impractical to use without having target labels. In comparison, the S. Only model is close to 60% for all target label free model selection criterions performing better than many pairs of methods and model selection criterions. We believe this finding is very important to the community.

That is why we proposed the use of a 100 random labeled target samples for model selection, which leads to only a few percentage drop of performance against ORACLE, instead of relying on a model selection criteria. The consistent and improved performance of this approached in comparison to the target label free model selection strategies outweights the cost of labeling 100 target samples.

Finally, our work is the first to propose a rigorous evaluation protocol by encouraging the use of the same experimental protocol over methods and random seeds. Indeed the variation of performance induced by the random seed is to the extent that carefully chosen seeds can potentially lead to SOTA performance. This highlights the importance of disclosing the seeds as we recommend and to evaluate existing methods under the same experimental protocol.

In highlight of the above, we hope the reviewer reconsiders their decision to reject our paper and raises their score as we believe that some merits of our work have been undervalued in the review. We believe that showing that target label free model selection strategies fail for PDA methods is of value to the community and can spark researchers to focus on model selection strategies that so far have been under invistigated. Moreover, it would encourage future researchers to follow a fair experimental protocol to evaluate their methods.

From: ICLR.cc/2023/Conference/Paper5333/Authors
title: Response Reviewer unvz Part 2/2
comment: > The writing needs to improve. The Tables are referenced but always placed on different pages. For example, Table 2 is referred in page 4 but placed on page 3, making it hard to read. The paper also has many typos, e.g., ‘that’ instead of ‘than’ in section 3.

Thank you for point out the different typos. We have made a careful read of our paper to correct them. Regarding the tables, we chose to have one Table per page to improve readibility, but it had that unintended consequence. We have moved some of them to correct this while preserving the readibility experience. in our updated manuscript.

> Many abbreviations lack definition and cause confusion. ‘AR’ in Table 5 stands for domain adaptation tasks and algorithms.

Thank you for pointing out the confusion with the AR abbreviation. We renamed the tasks from AC to A2C, AR to A2R and so on. We have reviewed the paper and did not find any other abbreviations that lack definition and caused confusion. Can the reviewer point out what other definitions which caused confusion so we can provide additional clarifications?

> In section 4.2, heuristic strategies for Hyper-parameter turning is not clearly described.

We have reviewed Section 4.2. Please let us know if additional clarifications are required regarding the heuristic strategy.

> And the author said, “we only consider the model at the end of training”, but should we use the model selection strategies?

Yes, model selection should still be used. We are simply highlighting that when choosing the hyper-parameters by training the models on the AC task for Office-Home and SR for VisDA, we discard the intermediate model checkpoints. The reason is that we are looking for hyper-parameters that do not lead to overfitting. Once the hyper-parameters are fixed, the models are then trained on all tasks and the model selection strategy is used again to choose the best intermediate check point instead of just taking the model at the end of training.

> In section 5.2, part of Model Selection Strategies, the authors give a conclusion that seems to be wrong “only the JUMBOT and SND pair performed reasonably well with respect to the JUMBOT and ORACLE pair on both datasets.” In Table 7, the JUMBOT and SND pair performs worse than the JUMBOT and ORACLE pair by a large margin. For instance, on OFFICE-HOME, the JUMBOT and SND pair reaches 72.29 accuracies, while the JUMBOT and ORACLE pair achieves 77.15 accuracies.

Thank you to the reviewer for pointing out this out. The conclusion that only one pair is performing well still holds, but with a different pair. We have removed the sentence and replaced it with the following:

`Only the (BA3US, ENT) pair achieved average task accuracies within 3 percentage points of its ORACLE counterpart (i.e., (BA3US, ORACLE)), while still improving over S. Only model.`


We hope to have addressed all your comments and concerns and we are happy to provide any further clarifications if needed. If the reviewer is convinced with our answer, we hope it leads to increasing the reviewer's score as we believe that some of the merits of our paper may have been overlooked.

[1] Gulrajani et al., In Search of Lost Domain Generalization

From: ICLR.cc/2023/Conference/Paper5333/Authors
title: Response Reviewer unvz Part 1/2
comment: We thank the reviewer for identifying and recognizing the significant value of our contributions in this work. Please find below our responses to the reviewer's questions, we hope this clarifies all the reviewer's concerns. We are happy to provide more feedback if necessary.

> This work raises an essential issue in partial domain adaptation and evaluates many PDA algorithms and model selection strategies.

Thank you again for highlighting the importance of work for the partial domain adaptation community. 

> However, it does not present any solution to this problem.

Thank you for giving us the opportunity to clarify the novelty of our work.

First, we would like to emphasize that the main purpose of our work was to perform an empirical study on the impact of model selection strategies on the performance of partial domain adaptation methods. We have updated the paper to make this contribution clearer. We believe to have successfully done so providing new insight to the community. Previous work on model selection focus mainly on standard unsupervised domain adaptation showing limited results on PDA tasks. You et al. 2019 use DEV to choose the best model along training using PADA but do not tune the hyper-parameters. Saito et al. (2021) present results on only 3 taks of Office-Home and do not use PDA specific methods. This is in contrast to our experiments that consider 7 different PDA methods and 7 different model selection strategies.

Second, we kindly ask the reviewer to elaborate on what a solution to this problem would be. Is a solution a model selection strategy that works well across all PDA methods and datasets? Our results suggest that that it is too much to ask given how different the performance of a PDA algorithm and model selection strategy pair can be across datasets (e.g. the M-POT method paired with SND provides the best performance on VisDA but the worst performance on Office-Home, while BA3US paired with SND provides the worst performance on VisDA but the best performance on Office-Home.) Instead, we argue that the model selection strategy is dependent on the PDA method and suggest that "for future proposed methods, researchers should specify [...] which model selection strategy should be used" (see end of Section 5.2). This finding is similar to [1] where a similar study was conducted for Domain Generalization.

Given the above, we therefore do not share the reviewer's opinion that the lack of a solution is a weakness of our work, being in fact out of scope.

>The findings of the experiments are a bit trivial. No target label for model selection strategies will hurt the performance, and the random seed would influence the performance. They are commonsense in domain adaptation, even in deep learning.

Thank you for giving us the opportunity to provide further clarifications on one of our main contributions. While it is natural to expect a drop of performance when target labels are not used for model selection, previous work claimed that target label free model selection strategies are still able to select the best models. For instance, Saito et al (2021) claimed that SND leads to performance close to optimal in 80% of the cases. However our study invalidates this claim. On Office-Home, the average task accuracy of M-POT paired with SND drops to 46.92% from 77.31% when paired with ORACLE. On the VisDA dataset, SND paired with either SAFN, BA3US and AR also perform very poorly in comparison to ORACLE. In fact, SND is the worst performing model selection strategy on VisDA for BA3US and AR methods. Such a drop in performance is significant and points towards PDA methods being impractical to use without having target labels. In comparison, the S. Only model is close to 60% on Office-Home regargless of the target label free model selection strategy used performing better or on par than many pairs of PDA methods and target label free model selection strategies. We believe this finding is very important to the community.

Regarding the random seed, it is natural that it influences the performance of algorithms. However, we are the first to analyse its effect and show that the variation of performance is to the extent that carefully chosen seeds can potentially lead to SOTA performance. This highlights the importance of disclosing the seeds as we recommend and to reproduce existing methods under the same experimental protocol.

From: ICLR.cc/2023/Conference/Paper5333/Authors
title: Response Reviewer PkyR
comment: We thank the reviewer for identifying and recognizing the significant value of our contributions in this work. Please find below our responses to the reviewer’s questions, which we hope clarifies all the reviewer’s concerns. We are happy to provide more feedback as necessary.

> While the paper does a lot of experiments, the paper does not establish a strong baseline.
 
First, we would like to emphasize that one of our contributions is precisely a fair experimental protocol that we hope researchers will adopt. As we highlight in our paper (see Table 2) the experimental protocols adopted in previous works exhibit differences that have a significant impact on the performance of the methods (the choice of bottleneck layer can lead to a 4 percentage point difference in performance), as well as design flaws (only 1 run for each task in some instances and the use of target labels during evaluation). There is therefore a need for a fair experimental protocol to be adopted by the community and our work proposes one.

Regarding the strong baseline, we infer that the reviewer refers to a domain adaptation method (please correct us if otherwise). In such case, the source only method fullfils the requirements since its performance remains consistent for all label-free model selection strategies on both datasets.

Furthermore, we believe we should look at PDA methods and model selections as pairs and the goal should not be to have a PDA method that works with all model selection strategies, neither a model selection strategy that works with all PDA methods. Instead, we believe that for practioners the value relies in having a PDA algorithm and model selection method that works well across *datasets* similarly as in Domain Generalization[1]. Our results show that is only true (to some extent) for the (BA3US, ENT) pair.

[1] Gulrajani et al., In Search of Lost Domain Generalization


> In Table 1 it is not clear how the authors got the "best" and "worst" performances. Please clarify.

The "best" and "worst" performance for each method correspond, respectively, to the maximum accuracy and minumum accuracy achieved by each method using one of the four model selection criterias that do not rely on target labels (S-ACC, ENT, DEV, SND). These can be found in Table 6 and in the updated version we highlight in green and red for improved readibility.

> How are the seeds chosen? Randomly?

Thank you for this question. The code from the official repositories for BA3US, AR, JUMBOT and M-POT all define a seed variable argument to run their code with a default value of 2020. We infered that this value refer to the calendar year as this was the year that BA3US was published. As we are currently in 2022, we made the simple choice to use 2020, 2021, 2022.

> What effect can the optimizer and learning rate strategy have on the algorithms? I suppose they would also have some dependency on it.

Thank you for the interesting question. We share the reviewer's curiosity and its opinion that it could affect the performance of the PDA methods. We chose not explore it in this work and leave it for future work as we find it to be orthogonal to our work regarding the impact of model selection strategies. Moreover, the optimizer and learning rate strategy we adopted is the same as the ones implemented in the different official code implementations.

> However, a simple strong baseline and a suggestion for a robust experimental protocol is missing, which should be the take-away from such works.

We kingly disagree with the reviewer here as we are in fact proposing a robust experimental protocol. We suggest that all future PDA methods should follow the same experimental protocol (i.e., architecture and training settings). We also encourage authors to specify a target label free model selection that should be used with the proposed method.


We hope to have addressed all your major comments. We are happy to provide any further clarifications. In light of our response, we hope the reviewer increases their rating.

From: ICLR.cc/2023/Conference/Paper5333/Reviewer_PkyR
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: This paper raises an important question about reproducibility of results for partial domain adaptation. The authors show that the stopping criterion is key to the success of partial domain adaptation algorithms in the literature, and there can be massive variations for various stopping criterions commonly used in the literature. Some algorithms even produce wide variations in performance for different random seeds. 
strength_and_weaknesses: Strength
- The paper have done considerable amount of experiments along various dimensions
- The paper is well-written and easy to follow

Weakness
- While the paper does a lot of experiments, the paper does not establish a strong baseline. The main takeaway from the paper is that partial domain adaptation works in literature produce high variations in performance to stopping criterions and random seeds. 
The expected takeaway from such papers should be rather a simple strong baseline which can be concluded from the experiments under a certain fair experimental protocol, which works well across stopping criterions and random seeds. This would then encourage researchers to follow that experimental protocol and report their results and compare against that baseline. 

The paper is overall in a good direction and I highly encourage the authors to consider this to refine their work. 
clarity,_quality,_novelty_and_reproducibility: - In Table 1 it is not clear how the authors got the "best" and "worst" performances. Please clarify.

- How are the seeds chosen? Randomly? 

- What effect can the optimizer and learning rate strategy have on the algorithms? I suppose they would also have some dependency on it.

- 
summary_of_the_review: The paper raises an interesting question, presents various experiments to showcase the problems with the current methods. However, a simple strong baseline and a suggestion for a robust experimental protocol is missing, which should be the take-away from such works. 
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5333/Reviewer_unvz
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: This paper presents a benchmark for partial domain adaptation (PDA) to evaluate the model selection strategies for different PDA approaches on two different real-world datasets. The experimental results show that 1) Target labels are critical for model selection strategies; 2) only one method and model selection pair perform well on both datasets; 3) The results of PDA methods are influenced by the rand seeds. The authors also give some recommendations for future works.
strength_and_weaknesses: Strength:
1.	This paper studies an important domain adaptation problem, particularly for partial domain adaptation. Using the target label to select models violates the domain adaptation assumption. But there is no suitable and effective metric for model selection. This is why researchers still use target labels for model selection.
2.	This paper evaluates 7 representative PDA algorithms on 2 different real-world datasets using 7 different model selection strategies. This benchmark will foster the research community.
3.	Based on the extensive experiments, the authors give some conclusions and suggestions for future works.

Weakness:
1.	This work raises an essential issue in partial domain adaptation and evaluates many PDA algorithms and model selection strategies. However, it does not present any solution to this problem.
2.	The findings of the experiments are a bit trivial. No target label for model selection strategies will hurt the performance, and the random seed would influence the performance. They are commonsense in domain adaptation, even in deep learning.
3.	The writing needs to improve. The Tables are referenced but always placed on different pages. For example, Table 2 is referred in page 4 but placed on page 3, making it hard to read. The paper also has many typos, e.g., ‘that’ instead of ‘than’ in section 3.
4.	Many abbreviations lack definition and cause confusion. ‘AR’ in Table 5 stands for domain adaptation tasks and algorithms.
5.	In section 4.2, heuristic strategies for Hyper-parameter turning is not clearly described. And the author said, “we only consider the model at the end of training”, but should we use the model selection strategies?
6.	In section 5.2, part of Model Selection Strategies, the authors give a conclusion that seems to be wrong “only the JUMBOT and SND pair performed reasonably well with respect to the JUMBOT and ORACLE pair on both datasets.” In Table 6, the JUMBOT and SND pair performs worse than the JUMBOT and ORACLE pair by a large margin. For instance, on OFFICE-HOME, the JUMBOT and SND pair reaches 72.29 accuracies, while the JUMBOT and ORACLE pair achieves 77.15 accuracies.
clarity,_quality,_novelty_and_reproducibility: The clarity, quality, and novelty are only marginally significant. For reproducibility, the authors provide the key resources to reproduce the results.
summary_of_the_review: The authors study an important issue in partial domain adaptation and conduct extensive experiments to benchmark different PDA methods and model selection strategies. Based on these experimental results, this paper shows some findings. However, this paper does not present solutions for this issue, and the findings are also obvious.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold

From: ICLR.cc/2023/Conference/Paper5333/Reviewer_ZmSX
confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: Existing algorithms for the problem of PDA(Partial Domain Adaptation) are considered and evaluated. While most previous successes have relied on target labels during training(model selection strategy), the availability of target domain labels is unrealistic. 

With a more realistic evaluation of the PDA methods, the authors claim that performance drops are significant without target labels. The finding urges the community to rethink the evaluation protocol for practical PDA methods. 
strength_and_weaknesses: As the study of PDA methods is relatively new to the community, the benchmark and evaluation protocol for the field still needs a careful study/investigation. The submitted paper has dealt with the problem of interest comprehensively and stated the PDA model selection checkpoints. This is an interesting and significant technical challenge that matters to the community. 

The benchmark and relevant code are open-sourced for reproducibility. 

While the paper is well-written overall, some sentences are difficult to follow. Some examples are as follows: 
p.2 protocal -> protocol, methods pairs -> method pairs, 100 random samples is -> 100 random samples are 
p.3 that distinguish -> that distinguishes, a high confident model -> a highly confident model
p.4  transform ... in a vanilla -> transform ... into a vanilla, found than target -> found that target.



clarity,_quality,_novelty_and_reproducibility: While the tackled problem of interest, a realistic evaluation of PDA methods, is original, the novelty of the work overall is still in question. The assessment and the reproduction of the existing methods are relatively straightforward, and the most significant claim that the absence of target labels can hurt the performance of the finally chosen PDA model is not surprising. 

The paper is challenging to follow and understand. More detailed analyses, explanations, and visualizations of the major claim and the pitfall of the existing algorithms would help the readers understand more clearly. 

The code is open-sourced for reproducibility. 

summary_of_the_review: While the paper is somewhat interesting, it does not suggest any interesting algorithm or an exciting finding. Most experiments are straightforward and there are not many insights the readers can learn from. 
The paper is not clear and is difficult to comprehend. 
For the aforementioned reasons(novelty, technical contribution, and clarity), I recommend rejecting it.
correctness: 4: All of the claims and statements are well-supported and correct.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough