Paper:
Unsupervised Domain Adaptation with Residual Transfer Networks
The recent success of deep neural networks relies on massive amounts of labeled data. For a target task where labeled data is unavailable, domain adaptation can transfer a learner from a different source domain. In this paper, we propose a new approach to domain adaptation in deep networks that can jointly learn adaptive classifiers and transferable features from labeled data in the source domain and unlabeled data in the target domain. We relax a shared-classifier assumption made by previous methods and assume that the source classifier and target classifier differ by a residual function. We enable classifier adaptation by plugging several layers into deep network to explicitly learn the residual function with reference to the target classifier. We fuse features of multiple layers with tensor product and embed them into reproducing kernel Hilbert spaces to match distributions for feature adaptation. The adaptation can be achieved in most feed-forward models by extending them with new residual layers and loss functions, which can be trained efficiently via back-propagation. Empirical evidence shows that the new approach outperforms state of the art methods on standard domain adaptation benchmarks.
by a small perturbation function. The goal of this paper is to simultaneously learn adaptive classifiers and transferable features from labeled data in the source domain and unlabeled data in the target domain by embedding the adaptations of both classifiers and features in a unified deep architecture.
Motivated by the state of the art deep residual learning [8], winner of the ImageNet ILSVRC 2015 challenge, we propose a new Residual Transfer Network (RTN) approach to domain adaptation in deep networks which can simultaneously learn adaptive classifiers and transferable features. We relax the shared-classifier assumption made by previous methods and assume that the source and target classifiers differ by a small residual function. We enable classifier adaptation by plugging several layers into deep networks to explicitly learn the residual function with reference to the target classifier. In this way, the source classifier and target classifier can be bridged tightly in the back-propagation procedure. The target classifier is tailored to the target data better by exploiting the low-density separation criterion. We fuse features of multiple layers with tensor product and embed them into reproducing kernel Hilbert spaces to match distributions for feature adaptation. The adaptation can be achieved in most feed-forward models by extending them with new residual layers and loss functions, and can be trained efficiently using standard back-propagation. Extensive evidence suggests that the RTN approach outperforms several state of art methods on standard domain adaptation benchmarks.

Review:
The paper presents a new deep learning architecture for the unsupervised adaptation problem, i.e. when one has labeled training data for a source domain and unlabeled data only for the target domain.  The approach mixes ideas of deep adaptation networks (ref. 5), residual networks (ref. 8) and entropy minimization for unsupervised learning (ref. 28) to propose an accurate and principled approach.   
The proposed architecture is a smart mix of previously published ideas : use of MK-MMD from deep adaptation networks (ref. 5), including residual networks blocks from (ref. 8) to relate the classifiers in the source domain and in the target domain, and entropy minimization (ref. 28) to guide the final learning in the target domain, where there are no labeled training samples. This  allows the authors outperforming state of the art methods on standard benchmarks.   The use of residual learning combined with entropy minimization learning criterion to jointly learn the source and target classifier is new and smart and it seems to work well in practice. Yet although the residual learning idea seems complementary to the MK-MMD of (ref. 5) (in the experimental section) this works looks more like an incremental work over (ref. 5) than a brand new model that is very different from (ref. 5) as the authors claim.    I don’t fully understand the difference between the use of MK MMD here with respect to what is proposed in (ref. 5). What is different ? Why is it better here ? And what is the motivation ? This part is then not fully convincing and especially since there is no experimental comparison with the method proposed in (ref. 5) and the variant proposed here.  It is said multiple times that the residual learning framework guarantees that the residual part will be small but i am  not sure there are such guarantees actually. It is more an experimental finding but in a very different setting. Why should it be true here ?  Experimental results are provided for the two  benchmarks of (ref. 5) and show quite convincing improvements of proposed method with respect to state of the art and put in evidence the actual complementarity of MMD criterion and of the residual learning idea. As said above a comparison with the same MMD use was in (ref. 5) would help appreciating if this variant is efficient or not here. There are a number of results which are different from the ones in (ref. 5) although the experimental protocol described in 4.1 looks strictly the same.  This is the case for DAN, the method in (ref. 5), which reaches 72.9 average accuracy on Office-31 dataset while it is reported at 70.0 here (ref. Table 1). Comments on such differences should be given. Moreover results from benchmark methods like TCA and GFK are much better here than results reported in (ref. 5) (~66% instead of ~27%)which might be due to the use of deep features as input where these methods were used on rough features in (ref. 5)? Please provide more precision on that.   Overall the paper is nice and easy to read. It proposes a new brick to previously designed deep architectures for unsupervised adaptation. It is then partially incremental in my opinion but still it is a nice use of residual learning idea.