  Unsupervised Domain Adaptation (UDA) transfers predictive models from a
fully-labeled source domain to an unlabeled target domain. In some
applications, however, it is expensive even to collect labels in the source
domain, making most previous works impractical. To cope with this problem,
recent work performed instance-wise cross-domain self-supervised learning,
followed by an additional fine-tuning stage. However, the instance-wise
self-supervised learning only learns and aligns low-level discriminative
features. In this paper, we propose an end-to-end Prototypical Cross-domain
Self-Supervised Learning (PCS) framework for Few-shot Unsupervised Domain
Adaptation (FUDA). PCS not only performs cross-domain low-level feature
alignment, but it also encodes and aligns semantic structures in the shared
embedding space across domains. Our framework captures category-wise semantic
structures of the data by in-domain prototypical contrastive learning; and
performs feature alignment through cross-domain prototypical self-supervision.
Compared with state-of-the-art methods, PCS improves the mean classification
accuracy over different domain pairs on FUDA by 10.5%, 3.5%, 9.0%, and 13.2% on
Office, Office-Home, VisDA-2017, and DomainNet, respectively. Our project page
is at http://xyue.io/pcs-fuda/index.html
