  Object recognition advances very rapidly these days. One challenge is to
generalize existing methods to new domains, to more classes and/or to new data
modalities. In order to avoid annotating one dataset for each of these new
cases, one needs to combine and reuse existing datasets that may belong to
different domains, have partial annotations, and/or have different data
modalities. This paper treats this task as a multi-source domain adaptation and
label unification (mDALU) problem and proposes a novel method for it. Our
method consists of a partially-supervised adaptation stage and a
fully-supervised adaptation stage. In the former, partial knowledge is
transferred from multiple source domains to the target domain and fused
therein. Negative transfer between unmatched label space is mitigated via three
new modules: domain attention, uncertainty maximization and attention-guided
adversarial alignment. In the latter, knowledge is transferred in the unified
label space after a label completion process with pseudo-labels. We verify the
method on three different tasks, image classification, 2D semantic image
segmentation, and joint 2D-3D semantic segmentation. Extensive experiments show
that our method outperforms all competing methods significantly.
