Paper:
This paper presents a generic framework to remedy the misalignment in unsupervised domain adaptation (UDA). Previous adversarial learning methods for UDA condition domain alignment only on pseudo labels, but pseudo labels may be inaccurate, hence bringing insufficient alleviation to the misalignment. Compared with pseudo labels, class prototypes are more accurate and reliable since they are summarized over all the instances and are able to represent the inherent semantic structures shared across domains. Therefore, we propose a novel Prototype-Assisted Adversarial Learning (PAAL) scheme, which incorporates instance probabilistic predictions and class prototypes together to provide reliable indicators for adversarial domain adaptation. With the PAAL scheme, we align both the instance feature representations and class prototypes to alleviate the misalignment among semantically different instances. Also, we exploit the class prototypes as proxy to minimize the intra-class variance in the target domain to mitigate the misalignment among semantically similar instances. With these novelties, we constitute a Prototype-Assisted Conditional Domain Adaptation (PACDA) framework which achieves state-of-the-art results on two UDA tasks, i.e., object recognition (OfficeHome, ImageCLEF-DA, and Office) and synthetic-to-real semantic segmentation (GTA5→Cityscapes and Synthia→Cityscapes).

Review:
Summary:
- key problem: address "class mismatch" in adversarial learning methods for unsupervised domain adaptation (UDA);
- contributions: 1) extension of the domain adversarial learning objective to leverage class prototypes (exponential moving average of features weighted by predicted class probabilities) in addition to pseudo-labels and intermediate representations (cf. eqs.5-11), 2) state-of-the-art results on several UDA tasks (Office-Home, ImageCLEF-DA, sim2real on Cityscapes).
Recommendation: weak accept (with some reservations below).
Key reason: interesting and effective use of prototypes for UDA.
- The formulation of the prototypes and additional learning objectives for UDA are clear and seem novel, although I would like to see a discussion of additional related works:
-- "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results", Tarvainen and Valpola, NeurIPS'17;
-- "Unsupervised Domain Adaptation with Similarity Learning", Pinheiro, CVPR'18;
-- "Transferable Prototypical Networks for Unsupervised Domain Adaptation", Pan et al, CVPR'19.
- The effectiveness of the contributions is validated on multiple UDA tasks, and the ablative analysis supports the claims (that prototype-level alignment and within-class compactness helps).
Main reservation: the specific problem is not clearly formalized.
- What is the often mentioned but not clearly described "class mismatch" problem in UDA? To the best of my knowledge, this not a standard problem (could not find any mention in the previous literature, no citations or definitions in the submission). Is it that the target label space is different than the source label space (e.g., different ontologies)? In this case, what is the information on the target label space that enables unsupervised adaptation from the source one? What is the inductive bias / prior / assumptions? 
- Alternatively, is the tackled problem only the noise in the pseudo-labels?
- In any case, the submission would greatly benefit from a clearer mathematical formalism and experimental characterization of the specific problem tackled here, especially in light of claims like "conditioning the alignment on pseudo labels can not well address the mismatch problem. Compared with the pseudo labels, the class prototypes are more robust and reliable in terms of representing the distribution of different semantic classes."
Additional Feedback:
- missing references on sim2real UDA: "DADA: Depth-aware Domain Adaptation in Semantic Segmentation" (Vu et al, ICCV'19), "SPIGAN: Privileged Adversarial Learning from Simulation" (Lee et al, ICLR'19)
## Post rebuttal update
I would like to thank the authors for replying to our questions. The clarifications with respect to related works and missing references is helpful, although a bit high-level (i.e. not necessarily describing the relative advantages of the proposed method). Nonetheless, the expected benefits of prototypes is still not entirely clear enough here, for instance regarding the main statistical assumptions that the method needs to make to get robust prototypes (e.g., in the presence of outliers or specific forms of "inaccuracies" in the pseudo-labels or "domain misalignment"). Therefore, due to the overall lack of mathematical clarity in the text and rebuttal, my main reservation remains, and I will change my "weak accept / borderline" score to weak reject. I encourage the authors to formalize the problem in a clearer, non-ambiguous way, discussing more explicitly the limitations of the proposed method.